[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:43 dist-test-slave-3wc3 krb5kdc[24252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:43 dist-test-slave-3wc3 krb5kdc[24252](info): set up 2 sockets
Sep 27 22:14:43 dist-test-slave-3wc3 krb5kdc[24252](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:14:44.386211 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:14:44 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:14:44.566227 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39297
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:44.894055 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:44.894356 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:44.894516 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:44.901769 24265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:44.901875 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:44.901970 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:44.902086 24265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:44.902168 24265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:44.912195 24265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:39297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:44.913547 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:44.914934 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:44.926973 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:44.927009 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:44.927043 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:44.928412 24265 server_base.cc:1034] running on GCE node
I20260927 22:14:44.929241 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:44.930744 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:44.931951 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790547284931879 us; error 58 us; skew 500 ppm
I20260927 22:14:44.932317 24265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:44 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547284, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:44.935528 24265 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:14:44.937026 24265 webserver.cc:460] Webserver started at http://127.23.173.126:33267/ using document root <none> and password file <none>
I20260927 22:14:44.937662 24265 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:44.937762 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:44.938082 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:44.940987 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "dc13f10bc34a47a0bc8deb55d21297b9"
format_stamp: "Formatted at 2026-09-27 22:14:44 on dist-test-slave-3wc3"
server_key: "2f0cabb4ea96bc9e596006d9a4128ca7"
server_key_iv: "ab6b14737b0dd8011d6d6317e3c3862b"
server_key_version: "encryptionkey@0"
I20260927 22:14:44.941677 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "dc13f10bc34a47a0bc8deb55d21297b9"
format_stamp: "Formatted at 2026-09-27 22:14:44 on dist-test-slave-3wc3"
server_key: "2f0cabb4ea96bc9e596006d9a4128ca7"
server_key_iv: "ab6b14737b0dd8011d6d6317e3c3862b"
server_key_version: "encryptionkey@0"
I20260927 22:14:44.946975 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:14:44.950645 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:44.952093 24265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:44.952277 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "dc13f10bc34a47a0bc8deb55d21297b9"
format_stamp: "Formatted at 2026-09-27 22:14:44 on dist-test-slave-3wc3"
server_key: "2f0cabb4ea96bc9e596006d9a4128ca7"
server_key_iv: "ab6b14737b0dd8011d6d6317e3c3862b"
server_key_version: "encryptionkey@0"
I20260927 22:14:44.952486 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:44.980654 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:44.985215 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:44.985519 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:45.004806 24265 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:39297
I20260927 22:14:45.004849 24331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:39297 every 8 connection(s)
I20260927 22:14:45.006207 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:45.010202 24332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:45.013043 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24265
I20260927 22:14:45.013397 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:14:45.016101 24245 external_mini_cluster.cc:1428] Setting key 0526819ec0bc96b4734a2cf38e38a68d
I20260927 22:14:45.024412 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9: Bootstrap starting.
I20260927 22:14:45.027561 24332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:45.028654 24332 log.cc:824] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:45.032217 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9: No bootstrap required, opened a new log
I20260927 22:14:45.042258 24332 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } }
I20260927 22:14:45.042770 24332 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:45.042889 24332 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc13f10bc34a47a0bc8deb55d21297b9, State: Initialized, Role: FOLLOWER
I20260927 22:14:45.043609 24332 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } }
I20260927 22:14:45.043889 24332 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:45.044055 24332 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:45.044252 24332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 22:14:45 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:45.047905 24332 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } }
I20260927 22:14:45.048478 24332 leader_election.cc:302] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc13f10bc34a47a0bc8deb55d21297b9; no voters:
I20260927 22:14:45.049124 24332 leader_election.cc:288] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:45.049289 24337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:45.050061 24337 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [term 1 LEADER]: Becoming Leader. State: Replica: dc13f10bc34a47a0bc8deb55d21297b9, State: Running, Role: LEADER
I20260927 22:14:45.050671 24337 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } }
I20260927 22:14:45.053452 24332 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:45.055012 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } } }
I20260927 22:14:45.055416 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:45.063398 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:45.021004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.23.173.126:39297)
0927 22:14:45.022124 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:45.022141 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:45.023384 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 22:14:45.032967 (+ 9583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:45.032998 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:45.033496 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:45.033943 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:45.036804 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.039148 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.040843 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.042051 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.042269 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:45.049620 (+ 7351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:45.049636 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:45.049659 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:45.049721 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:45.054651 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:45.055800 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:45.055812 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:45.055824 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:45.056086 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:45.056765 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:45.056773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:45.056779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:45.057695 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:45.058084 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:45.061109 (+ 3025us) server_negotiation.cc:299] Negotiation successful
0927 22:14:45.061631 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":518,"threads_started":1}
I20260927 22:14:45.064441 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc13f10bc34a47a0bc8deb55d21297b9. Latest consensus state: current_term: 1 leader_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13f10bc34a47a0bc8deb55d21297b9" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39297 } } }
I20260927 22:14:45.064718 24342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:45.064787 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:45.072726 24342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:45.082640 24342 catalog_manager.cc:1347] Generated new cluster ID: 85bb7be18a97400e8e9953609913f392
I20260927 22:14:45.082759 24342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:45.095937 24342 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:45.097705 24342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:45.115554 24342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc13f10bc34a47a0bc8deb55d21297b9: Generated new TSK 0
I20260927 22:14:45.116784 24342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:14:45.170588 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:45.496799 24359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:45.497114 24359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:45.497291 24359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:45.504523 24359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:45.504673 24359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:45.504871 24359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:14:45.514310 24359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:45.515561 24359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:45.516897 24359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:45.529912 24364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:45.529874 24365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:45.529937 24367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:45.530802 24359 server_base.cc:1034] running on GCE node
I20260927 22:14:45.531391 24359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:45.532266 24359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:45.533499 24359 hybrid_clock.cc:648] HybridClock initialized: now 1790547285533457 us; error 54 us; skew 500 ppm
I20260927 22:14:45.533839 24359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:45 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547285, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:45.536926 24359 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:14:45.538309 24359 webserver.cc:460] Webserver started at http://127.23.173.65:45887/ using document root <none> and password file <none>
I20260927 22:14:45.538919 24359 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:45.539017 24359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:45.539253 24359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:45.542038 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "99544ceab9c44d7dbd9ad3ee23d7d98d"
format_stamp: "Formatted at 2026-09-27 22:14:45 on dist-test-slave-3wc3"
server_key: "94c800c926fe62fb84ff4a0bc9c68991"
server_key_iv: "19c01b4c9f12a276ba65f2aacbd7c751"
server_key_version: "encryptionkey@0"
I20260927 22:14:45.542742 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "99544ceab9c44d7dbd9ad3ee23d7d98d"
format_stamp: "Formatted at 2026-09-27 22:14:45 on dist-test-slave-3wc3"
server_key: "94c800c926fe62fb84ff4a0bc9c68991"
server_key_iv: "19c01b4c9f12a276ba65f2aacbd7c751"
server_key_version: "encryptionkey@0"
I20260927 22:14:45.547787 24359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:45.551163 24374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:45.552646 24359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:45.552883 24359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "99544ceab9c44d7dbd9ad3ee23d7d98d"
format_stamp: "Formatted at 2026-09-27 22:14:45 on dist-test-slave-3wc3"
server_key: "94c800c926fe62fb84ff4a0bc9c68991"
server_key_iv: "19c01b4c9f12a276ba65f2aacbd7c751"
server_key_version: "encryptionkey@0"
I20260927 22:14:45.553098 24359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:45.568935 24359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:45.573395 24359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:45.573635 24359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:45.574594 24359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:45.575871 24359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:45.575943 24359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:45.576027 24359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:45.576074 24359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:45.610844 24359 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:33009
I20260927 22:14:45.610891 24486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:33009 every 8 connection(s)
I20260927 22:14:45.612169 24359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:14:45.618531 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24359
I20260927 22:14:45.618777 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:14:45.619379 24245 external_mini_cluster.cc:1428] Setting key bee22ae30cd448d1aed56021e3eca3bb
Sep 27 22:14:45 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547285, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:45.632550 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:45.614791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:56817 (local address 127.23.173.126:39297)
0927 22:14:45.615321 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:45.615325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:45.616391 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 22:14:45.618446 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:45.618465 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:45.618564 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:45.618698 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:45.620368 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.621064 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.622731 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.623150 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.623295 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:45.626926 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:45.626931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:45.626934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:45.626964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:45.629469 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:45.630242 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:45.630250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:45.630258 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:45.630370 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:45.630845 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:45.630850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:45.630853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:45.631130 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:45.631299 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:45.631964 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 22:14:45.632179 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"spinlock_wait_cycles":3840,"thread_start_us":293,"threads_started":1}
I20260927 22:14:45.633523 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:45.615411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39297 (local address 127.23.173.65:56817)
0927 22:14:45.616224 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:45.616275 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:45.618040 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:45.618959 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:45.618968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:45.619508 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:45.620193 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:45.620225 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.621243 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.621248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:45.622577 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:45.622589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:45.623380 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:45.623388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:45.623620 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:45.624319 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:45.624340 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:45.626708 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:45.629612 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:45.629621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:45.629636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:45.630122 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:45.630491 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:45.630497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:45.630500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:45.630730 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:45.631266 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:45.631274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:45.631602 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:14:45.631854 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:14:45.632363 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":362,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:14:45.635058 24487 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39297
I20260927 22:14:45.635403 24487 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:45.636206 24487 heartbeater.cc:511] Master 127.23.173.126:39297 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:14:45.638518 24297 ts_manager.cc:194] Registered new tserver with Master: 99544ceab9c44d7dbd9ad3ee23d7d98d (127.23.173.65:33009)
I20260927 22:14:45.639962 24297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:56817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:14:45.657637 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:45.983011 24493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:45.983304 24493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:45.983480 24493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:45.990850 24493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:45.990955 24493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:45.991159 24493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:14:46.000742 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:46.002100 24493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:46.003456 24493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:46.016934 24498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:46.016872 24501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:46.016909 24499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:46.018256 24493 server_base.cc:1034] running on GCE node
I20260927 22:14:46.018842 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:46.019789 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:46.021026 24493 hybrid_clock.cc:648] HybridClock initialized: now 1790547286020988 us; error 28 us; skew 500 ppm
I20260927 22:14:46.021341 24493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:46.024518 24493 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:14:46.026081 24493 webserver.cc:460] Webserver started at http://127.23.173.66:46213/ using document root <none> and password file <none>
I20260927 22:14:46.026750 24493 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:46.026925 24493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:46.027245 24493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:46.030138 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "55cd376303e44b9985e82a0855acc5d5"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "38738aed9161f343f9f968a69f82cf6e"
server_key_iv: "9cb3bf3e9955dc884cddf518abaf0966"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.030890 24493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "55cd376303e44b9985e82a0855acc5d5"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "38738aed9161f343f9f968a69f82cf6e"
server_key_iv: "9cb3bf3e9955dc884cddf518abaf0966"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.035954 24493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:46.039523 24508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.040809 24493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:46.040995 24493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "55cd376303e44b9985e82a0855acc5d5"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "38738aed9161f343f9f968a69f82cf6e"
server_key_iv: "9cb3bf3e9955dc884cddf518abaf0966"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.041214 24493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:46.076048 24493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:46.080271 24493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:46.080574 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:46.081542 24493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:46.083045 24493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:46.083154 24493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.083305 24493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:46.083381 24493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.117823 24493 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:37169
I20260927 22:14:46.117880 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:37169 every 8 connection(s)
I20260927 22:14:46.119277 24493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:14:46.124374 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24493
I20260927 22:14:46.124570 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:14:46.125088 24245 external_mini_cluster.cc:1428] Setting key 1259a0c7bb4bd969d3d3428cb5a8e544
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:14:46.142195 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.121916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34697 (local address 127.23.173.126:39297)
0927 22:14:46.122118 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.122123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.123508 (+ 1385us) server_negotiation.cc:407] Connection header received
0927 22:14:46.125983 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.125990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.126084 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.126197 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.127956 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.128694 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.131339 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.131777 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.131958 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.136076 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.136082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.136085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.136119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.139053 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.139799 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.139807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.139810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.139956 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.140401 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.140411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.140413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.140677 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.140843 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.141554 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.141810 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:14:46.143277 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.122574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39297 (local address 127.23.173.66:34697)
0927 22:14:46.123331 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:46.123370 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:46.125620 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:46.126448 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:46.126456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:46.126978 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:46.127745 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:46.127770 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.128925 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.128933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:46.131167 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:46.131185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.132051 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.132058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:46.132300 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.133066 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:46.133105 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:46.135849 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:46.139201 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:46.139210 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:46.139228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:46.139659 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:46.140085 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:46.140089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:46.140093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:46.140276 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:46.140819 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:46.140828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:46.141174 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 22:14:46.141470 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 22:14:46.142079 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":319,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
I20260927 22:14:46.145043 24621 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39297
I20260927 22:14:46.145390 24621 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:46.146313 24621 heartbeater.cc:511] Master 127.23.173.126:39297 requested a full tablet report, sending...
I20260927 22:14:46.147801 24297 ts_manager.cc:194] Registered new tserver with Master: 55cd376303e44b9985e82a0855acc5d5 (127.23.173.66:37169)
I20260927 22:14:46.148622 24297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:34697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:14:46.170411 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:46.493420 24626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:46.493728 24626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:46.493831 24626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:46.501576 24626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:46.501693 24626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:46.501850 24626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:14:46.511794 24626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:46.513031 24626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:46.514343 24626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:46.527884 24631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:46.527848 24632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:46.527925 24634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:46.528998 24626 server_base.cc:1034] running on GCE node
I20260927 22:14:46.529680 24626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:46.530646 24626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:46.531867 24626 hybrid_clock.cc:648] HybridClock initialized: now 1790547286531813 us; error 43 us; skew 500 ppm
I20260927 22:14:46.532187 24626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:46.535352 24626 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:14:46.537010 24626 webserver.cc:460] Webserver started at http://127.23.173.67:37507/ using document root <none> and password file <none>
I20260927 22:14:46.537715 24626 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:46.537874 24626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:46.538240 24626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:46.541235 24626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "4ce99e4a2fb14635a47033791e161e03"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "e0e73a6319c0409006197ed419ef6c1b"
server_key_iv: "dd72e5fcde17dfe62fafd7b94aa2f4fd"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.542096 24626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "4ce99e4a2fb14635a47033791e161e03"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "e0e73a6319c0409006197ed419ef6c1b"
server_key_iv: "dd72e5fcde17dfe62fafd7b94aa2f4fd"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.547813 24626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:46.551499 24641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.552976 24626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:14:46.553164 24626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "4ce99e4a2fb14635a47033791e161e03"
format_stamp: "Formatted at 2026-09-27 22:14:46 on dist-test-slave-3wc3"
server_key: "e0e73a6319c0409006197ed419ef6c1b"
server_key_iv: "dd72e5fcde17dfe62fafd7b94aa2f4fd"
server_key_version: "encryptionkey@0"
I20260927 22:14:46.553406 24626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:46.570254 24626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:46.574654 24626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:46.574967 24626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:46.575990 24626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:46.577317 24626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:46.577394 24626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.577538 24626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:46.577590 24626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:46.612668 24626 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:36225
I20260927 22:14:46.612691 24753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:36225 every 8 connection(s)
I20260927 22:14:46.613945 24626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:14:46.618652 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24626
I20260927 22:14:46.618854 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:14:46.619315 24245 external_mini_cluster.cc:1428] Setting key cacd104933ea6aba2c3354fe33c54631
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:46.635845 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.616840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34961 (local address 127.23.173.126:39297)
0927 22:14:46.617052 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.617056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.618286 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 22:14:46.620384 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.620390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.620533 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.620659 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.623018 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.623899 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.625512 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.626062 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.626158 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.629705 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.629709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.629713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.629751 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.632485 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.633190 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.633193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.633196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.633317 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.633770 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.633776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.633780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.634235 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.634405 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.635232 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.635521 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:14:46.636847 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.617265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39297 (local address 127.23.173.67:34961)
0927 22:14:46.618101 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:46.618138 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:46.620059 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:46.621065 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:46.621075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:46.621681 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:46.622734 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:46.622778 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.624055 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.624059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:46.625377 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:46.625389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.626265 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.626271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:46.626467 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.627141 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:46.627164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:46.629516 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:46.632615 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:46.632621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:46.632633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:46.633078 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:46.633419 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:46.633422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:46.633425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:46.633608 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:46.634403 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:46.634411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:46.634869 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 22:14:46.635146 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 22:14:46.635645 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":277,"threads_started":1}
I20260927 22:14:46.638263 24754 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39297
I20260927 22:14:46.638521 24754 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:46.639170 24754 heartbeater.cc:511] Master 127.23.173.126:39297 requested a full tablet report, sending...
I20260927 22:14:46.640468 24297 ts_manager.cc:194] Registered new tserver with Master: 4ce99e4a2fb14635a47033791e161e03 (127.23.173.67:36225)
I20260927 22:14:46.641698 24297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:34961
I20260927 22:14:46.642763 24487 heartbeater.cc:503] Master 127.23.173.126:39297 was elected leader, sending a full tablet report...
I20260927 22:14:46.648437 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:46.694111 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.678671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.23.173.65:33009)
0927 22:14:46.679375 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.679386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.679419 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:14:46.679585 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.679592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.679821 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.680000 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.681445 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.682578 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.683868 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.684441 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.684592 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.688627 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.688637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.688652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.688685 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.691723 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.692311 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.692317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.692320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.692430 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.692747 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.692750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.692752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.693115 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.693329 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.693536 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.693721 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":355,"threads_started":1}
W20260927 22:14:46.695346 24402 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36808
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:46.739583 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.725739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.23.173.65:33009)
0927 22:14:46.726007 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.726017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.726390 (+ 373us) server_negotiation.cc:407] Connection header received
0927 22:14:46.726539 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.726543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.726610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.726702 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.728053 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.728732 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.729907 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.730377 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.730468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.734349 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.734353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.734356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.734386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.736934 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.737650 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.737654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.737686 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.737849 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.738278 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.738281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.738284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.738538 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.738712 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.739030 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.739190 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:14:46.741204 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24359
I20260927 22:14:46.754643 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.742374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.23.173.65:33009)
0927 22:14:46.742617 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.742624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.742919 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:14:46.743104 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.743110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.743175 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.743302 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.744464 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.745271 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.746635 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.747056 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.747192 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.749454 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.749465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.749468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.749497 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.752000 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.752651 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.752656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.752659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.752796 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.753176 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.753180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.753183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.753438 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.753729 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.754046 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.754293 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:14:46.755448 24482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:46.842350 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24359
I20260927 22:14:46.859668 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24493
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:14:46.876444 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.860573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.23.173.66:37169)
0927 22:14:46.861113 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.861120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.861145 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:14:46.861207 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.861213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.861425 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.861554 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.862889 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.863936 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.865261 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.865840 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.865935 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.870292 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.870311 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.870327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.870361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.873360 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.874258 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.874268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.874271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.874413 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.874834 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.874838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.874840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.875253 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.875506 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.875815 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.876011 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":286,"threads_started":1}
I20260927 22:14:46.877751 24616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:46.960469 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24493
I20260927 22:14:46.976438 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24626
Sep 27 22:14:46 dist-test-slave-3wc3 krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:14:46.994729 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:46.977498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.23.173.67:36225)
0927 22:14:46.978247 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:46.978255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:46.978285 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:14:46.978352 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:46.978359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:46.978575 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:46.978772 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:46.980275 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.981485 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.983131 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:46.983736 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:46.983916 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:46.989048 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:46.989059 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:46.989074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:46.989108 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:46.992052 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.992763 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.992770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.992772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.992887 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:46.993259 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:46.993262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:46.993265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:46.993606 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:46.993881 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:46.994252 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 22:14:46.994448 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":332,"threads_started":1}
I20260927 22:14:46.996145 24749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:47.080255 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24626
I20260927 22:14:47.096609 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24265
I20260927 22:14:47.097856 24327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:47.202927 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24265
2026-09-27T22:14:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3921 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:47 dist-test-slave-3wc3 krb5kdc[24775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:47 dist-test-slave-3wc3 krb5kdc[24775](info): set up 2 sockets
Sep 27 22:14:47 dist-test-slave-3wc3 krb5kdc[24775](info): commencing operation
krb5kdc: starting...
W20260927 22:14:48.301890 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:14:48 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:14:48.483873 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34223
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:48.798252 24788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:48.798626 24788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:48.798785 24788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:48.806087 24788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:48.806217 24788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:48.806329 24788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:48.806424 24788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:48.806509 24788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:48.816227 24788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:48.817577 24788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:48.818863 24788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:48.830353 24793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:48.830402 24796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:48.830397 24794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:48.831094 24788 server_base.cc:1034] running on GCE node
I20260927 22:14:48.831775 24788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:48.833065 24788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:48.834255 24788 hybrid_clock.cc:648] HybridClock initialized: now 1790547288834207 us; error 36 us; skew 500 ppm
I20260927 22:14:48.834589 24788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:48 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547288, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:48.847783 24788 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:14:48.849397 24788 webserver.cc:460] Webserver started at http://127.23.173.126:37595/ using document root <none> and password file <none>
I20260927 22:14:48.850183 24788 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:48.850322 24788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:48.850634 24788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:48.853574 24788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c122b0c799304aafa11b1cb68dae83d8"
format_stamp: "Formatted at 2026-09-27 22:14:48 on dist-test-slave-3wc3"
server_key: "ef9aca54c7837799438327e62598a46a"
server_key_iv: "7b98492ae3201ac26b95241e0e17575a"
server_key_version: "encryptionkey@0"
I20260927 22:14:48.854341 24788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c122b0c799304aafa11b1cb68dae83d8"
format_stamp: "Formatted at 2026-09-27 22:14:48 on dist-test-slave-3wc3"
server_key: "ef9aca54c7837799438327e62598a46a"
server_key_iv: "7b98492ae3201ac26b95241e0e17575a"
server_key_version: "encryptionkey@0"
I20260927 22:14:48.859417 24788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:48.862773 24803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:48.864090 24788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:48.864270 24788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c122b0c799304aafa11b1cb68dae83d8"
format_stamp: "Formatted at 2026-09-27 22:14:48 on dist-test-slave-3wc3"
server_key: "ef9aca54c7837799438327e62598a46a"
server_key_iv: "7b98492ae3201ac26b95241e0e17575a"
server_key_version: "encryptionkey@0"
I20260927 22:14:48.864462 24788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:48.883846 24788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:48.888233 24788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:48.888520 24788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:48.906680 24788 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:34223
I20260927 22:14:48.906713 24854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:34223 every 8 connection(s)
I20260927 22:14:48.908032 24788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:48.909842 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24788
I20260927 22:14:48.910130 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:14:48.910986 24245 external_mini_cluster.cc:1428] Setting key c5b0e07eeda95db369a90dcc0fb28e40
I20260927 22:14:48.912611 24855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:14:48 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:48.926357 24855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8: Bootstrap starting.
I20260927 22:14:48.929595 24855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:48.930634 24855 log.cc:824] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:48.933213 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:48.912983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.23.173.126:34223)
0927 22:14:48.913719 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:48.913730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:48.913769 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:14:48.915395 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:48.915416 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:48.915799 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:48.916131 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:48.917912 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:48.919268 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:48.921075 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:48.921692 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:48.921870 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:48.926386 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:48.926394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:48.926408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:48.926447 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:48.929370 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:48.930094 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:48.930101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:48.930106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:48.930256 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:48.930605 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:48.930611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:48.930614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:48.931239 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:48.931580 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:48.931949 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:14:48.932333 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":347,"threads_started":1}
I20260927 22:14:48.933492 24855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8: No bootstrap required, opened a new log
I20260927 22:14:48.940382 24855 raft_consensus.cc:348] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } }
I20260927 22:14:48.940738 24855 raft_consensus.cc:374] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:48.940834 24855 raft_consensus.cc:729] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c122b0c799304aafa11b1cb68dae83d8, State: Initialized, Role: FOLLOWER
I20260927 22:14:48.941695 24855 consensus_queue.cc:260] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } }
I20260927 22:14:48.942093 24860 raft_consensus.cc:482] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:48.942317 24860 raft_consensus.cc:504] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } }
I20260927 22:14:48.942063 24855 raft_consensus.cc:388] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:48.942831 24855 raft_consensus.cc:482] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:48.942777 24860 leader_election.cc:302] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c122b0c799304aafa11b1cb68dae83d8; no voters:
I20260927 22:14:48.942963 24855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:48.943583 24860 leader_election.cc:288] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 22:14:48.945192 24855 raft_consensus.cc:504] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } }
I20260927 22:14:48.945447 24855 leader_election.cc:302] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c122b0c799304aafa11b1cb68dae83d8; no voters:
I20260927 22:14:48.945473 24861 raft_consensus.cc:2741] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 22:14:48.945721 24855 leader_election.cc:288] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:48.945811 24860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:48.946780 24860 raft_consensus.cc:686] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [term 1 LEADER]: Becoming Leader. State: Replica: c122b0c799304aafa11b1cb68dae83d8, State: Running, Role: LEADER
I20260927 22:14:48.947283 24860 consensus_queue.cc:237] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } }
I20260927 22:14:48.948294 24855 sys_catalog.cc:564] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:48.949261 24863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c122b0c799304aafa11b1cb68dae83d8. Latest consensus state: current_term: 1 leader_uuid: "c122b0c799304aafa11b1cb68dae83d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } } }
I20260927 22:14:48.949296 24861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c122b0c799304aafa11b1cb68dae83d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c122b0c799304aafa11b1cb68dae83d8" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34223 } } }
I20260927 22:14:48.949713 24861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:48.949709 24863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:48.952450 24866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:48.957410 24866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:48.965206 24866 catalog_manager.cc:1347] Generated new cluster ID: a0d632e924494eba9d909c1d49d95667
I20260927 22:14:48.965310 24866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:49.001706 24866 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:49.003090 24866 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:49.010146 24866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c122b0c799304aafa11b1cb68dae83d8: Generated new TSK 0
I20260927 22:14:49.011008 24866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:14:49.070313 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:49.392200 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:49.392485 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:49.392582 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:49.399706 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:49.399812 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:49.399930 24882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:14:49.409430 24882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:49.410671 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:49.412096 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:49.425228 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:49.425228 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:49.425252 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:49.426059 24882 server_base.cc:1034] running on GCE node
I20260927 22:14:49.426630 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:49.427515 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:49.428742 24882 hybrid_clock.cc:648] HybridClock initialized: now 1790547289428694 us; error 56 us; skew 500 ppm
I20260927 22:14:49.429071 24882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:49 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547289, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:49.432399 24882 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:14:49.434062 24882 webserver.cc:460] Webserver started at http://127.23.173.65:44265/ using document root <none> and password file <none>
I20260927 22:14:49.434762 24882 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:49.434890 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:49.435204 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:49.438130 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "5bebf4873d034dbb9dc3c06969138f07"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "e5d500694537a09b11371b14c9463ca0"
server_key_iv: "eed43ebd415bd601c03fec26c5dde038"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.438973 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "5bebf4873d034dbb9dc3c06969138f07"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "e5d500694537a09b11371b14c9463ca0"
server_key_iv: "eed43ebd415bd601c03fec26c5dde038"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.444104 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:49.447705 24897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:49.449111 24882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:49.449273 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "5bebf4873d034dbb9dc3c06969138f07"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "e5d500694537a09b11371b14c9463ca0"
server_key_iv: "eed43ebd415bd601c03fec26c5dde038"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.449476 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:49.474053 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:49.478071 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:49.478369 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:49.479312 24882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:49.480592 24882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:49.480691 24882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:49.480850 24882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:49.480942 24882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:49.512064 24882 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:38465
I20260927 22:14:49.512130 25009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:38465 every 8 connection(s)
I20260927 22:14:49.513360 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:14:49.519114 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 24882
I20260927 22:14:49.519326 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:14:49.520117 24245 external_mini_cluster.cc:1428] Setting key cfff2a436f1d8ab13b1d313ee36c168a
Sep 27 22:14:49 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547289, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:49.536072 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:49.515768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:50625 (local address 127.23.173.126:34223)
0927 22:14:49.516186 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:49.516192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:49.517134 (+ 942us) server_negotiation.cc:407] Connection header received
0927 22:14:49.519178 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:49.519197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:49.519294 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:49.519428 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:49.521339 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:49.522529 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:49.525044 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:49.525447 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:49.525546 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:49.530181 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:49.530186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:49.530189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:49.530219 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:49.532707 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:49.533680 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:49.533684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:49.533686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:49.533792 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:49.534357 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:49.534362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:49.534366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:49.534641 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:49.534759 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:49.535672 (+ 913us) server_negotiation.cc:299] Negotiation successful
0927 22:14:49.535828 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":194,"threads_started":1}
I20260927 22:14:49.537819 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:49.516331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34223 (local address 127.23.173.65:50625)
0927 22:14:49.516987 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:49.517023 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:49.518872 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:49.519671 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:49.519682 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:49.520419 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:49.521154 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:49.521177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:49.522683 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:49.522688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:49.524872 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:49.524889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:49.525663 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:49.525673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:49.526020 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:49.526786 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:49.526818 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:49.529946 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:49.532850 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:49.532862 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:49.532882 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:49.533570 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:49.533906 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:49.533914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:49.533922 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:49.534225 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:49.534791 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:49.534803 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:49.535249 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 22:14:49.535599 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 22:14:49.536254 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:14:49.539472 25010 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34223
I20260927 22:14:49.539760 25010 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:49.540457 25010 heartbeater.cc:511] Master 127.23.173.126:34223 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:14:49.542650 24820 ts_manager.cc:194] Registered new tserver with Master: 5bebf4873d034dbb9dc3c06969138f07 (127.23.173.65:38465)
I20260927 22:14:49.544785 24820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:50625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:14:49.564846 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:49.880841 25016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:49.881106 25016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:49.881263 25016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:49.888382 25016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:49.888485 25016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:49.888684 25016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:14:49.898979 25016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:49.900272 25016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:49.901573 25016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:49.914858 25022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:49.914954 25024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:49.915526 25021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:49.919507 25016 server_base.cc:1034] running on GCE node
I20260927 22:14:49.920132 25016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:49.921121 25016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:49.922382 25016 hybrid_clock.cc:648] HybridClock initialized: now 1790547289922322 us; error 50 us; skew 500 ppm
I20260927 22:14:49.922716 25016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:49 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547289, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:49.925824 25016 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:14:49.927367 25016 webserver.cc:460] Webserver started at http://127.23.173.66:45351/ using document root <none> and password file <none>
I20260927 22:14:49.928023 25016 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:49.928124 25016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:49.928372 25016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:49.931200 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "cfb4fbd0ff694dc5970aaf8306100289"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "212f7eae1cf3072f942d2e065cd9c94b"
server_key_iv: "37fa555323044837cfcc2df345e5fd2a"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.931869 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "cfb4fbd0ff694dc5970aaf8306100289"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "212f7eae1cf3072f942d2e065cd9c94b"
server_key_iv: "37fa555323044837cfcc2df345e5fd2a"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.936554 25016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:14:49.939745 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:49.940963 25016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:14:49.941087 25016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "cfb4fbd0ff694dc5970aaf8306100289"
format_stamp: "Formatted at 2026-09-27 22:14:49 on dist-test-slave-3wc3"
server_key: "212f7eae1cf3072f942d2e065cd9c94b"
server_key_iv: "37fa555323044837cfcc2df345e5fd2a"
server_key_version: "encryptionkey@0"
I20260927 22:14:49.941249 25016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:49.965221 25016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:49.969401 25016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:49.969653 25016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:49.970638 25016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:49.971908 25016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:49.971980 25016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:49.972059 25016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:49.972101 25016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:50.004635 25016 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34717
I20260927 22:14:50.004724 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34717 every 8 connection(s)
I20260927 22:14:50.005942 25016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:14:50.011786 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25016
I20260927 22:14:50.011998 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:14:50.012599 24245 external_mini_cluster.cc:1428] Setting key 0b05548436d92d05be07042c76f3e361
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547289, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:50.029462 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.008663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:60131 (local address 127.23.173.126:34223)
0927 22:14:50.008846 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.008850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.010130 (+ 1280us) server_negotiation.cc:407] Connection header received
0927 22:14:50.012302 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.012307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.012379 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.012468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.014315 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.015140 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.017794 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.018279 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.018380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.022294 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.022303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.022311 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.022357 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.025998 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.026700 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.026708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.026715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.026905 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.027397 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.027405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.027411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.027869 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.028028 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.028705 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.028989 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:50.030112 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.009218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34223 (local address 127.23.173.66:60131)
0927 22:14:50.009921 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.010008 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.011979 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.012745 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.012757 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.013455 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:50.014158 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.014179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.015317 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.015324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.017598 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.017617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.018498 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.018507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.018846 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.019609 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:50.019645 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:50.022053 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:50.026123 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:50.026129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:50.026144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:50.026559 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:50.027031 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:50.027036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:50.027040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:50.027267 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:50.028001 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:50.028007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:50.028314 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.028548 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.028968 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260927 22:14:50.031616 25144 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34223
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:14:50.031951 25144 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:50.032680 25144 heartbeater.cc:511] Master 127.23.173.126:34223 requested a full tablet report, sending...
I20260927 22:14:50.034215 24820 ts_manager.cc:194] Registered new tserver with Master: cfb4fbd0ff694dc5970aaf8306100289 (127.23.173.66:34717)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
I20260927 22:14:50.035138 24820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:60131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:14:50.063166 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:50.375831 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:50.376117 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:50.376283 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:50.383481 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:50.383584 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:50.383767 25149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:14:50.393183 25149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:50.394520 25149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:50.395747 25149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:50.408366 25155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:50.408370 25154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:50.408565 25157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:50.408968 25149 server_base.cc:1034] running on GCE node
I20260927 22:14:50.409703 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:50.410697 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:50.411916 25149 hybrid_clock.cc:648] HybridClock initialized: now 1790547290411868 us; error 47 us; skew 500 ppm
I20260927 22:14:50.412222 25149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:50.415367 25149 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:14:50.416671 25149 webserver.cc:460] Webserver started at http://127.23.173.67:38821/ using document root <none> and password file <none>
I20260927 22:14:50.417297 25149 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:50.417394 25149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:50.417711 25149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:50.420526 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "fcfc9c3d40b04de0bed6fac38fd9b820"
format_stamp: "Formatted at 2026-09-27 22:14:50 on dist-test-slave-3wc3"
server_key: "0bb154abf352998ded5280bcd0dbbfdb"
server_key_iv: "53c577da35be2006d7238ca1d9901f03"
server_key_version: "encryptionkey@0"
I20260927 22:14:50.421293 25149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "fcfc9c3d40b04de0bed6fac38fd9b820"
format_stamp: "Formatted at 2026-09-27 22:14:50 on dist-test-slave-3wc3"
server_key: "0bb154abf352998ded5280bcd0dbbfdb"
server_key_iv: "53c577da35be2006d7238ca1d9901f03"
server_key_version: "encryptionkey@0"
I20260927 22:14:50.426265 25149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:14:50.429821 25164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:50.431294 25149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:50.431484 25149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fcfc9c3d40b04de0bed6fac38fd9b820"
format_stamp: "Formatted at 2026-09-27 22:14:50 on dist-test-slave-3wc3"
server_key: "0bb154abf352998ded5280bcd0dbbfdb"
server_key_iv: "53c577da35be2006d7238ca1d9901f03"
server_key_version: "encryptionkey@0"
I20260927 22:14:50.431710 25149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:50.466459 25149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:50.471251 25149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:50.471552 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:50.472506 25149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:14:50.473814 25149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:50.473884 25149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:50.474082 25149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:50.474146 25149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:50.507436 25149 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:41543
I20260927 22:14:50.507531 25276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:41543 every 8 connection(s)
I20260927 22:14:50.508792 25149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:14:50.509274 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25149
I20260927 22:14:50.509526 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:14:50.510125 24245 external_mini_cluster.cc:1428] Setting key 219b7e81d978b3a7c778aa96faf195f1
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:50.530936 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.511835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:57385 (local address 127.23.173.126:34223)
0927 22:14:50.512045 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.512052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.513425 (+ 1373us) server_negotiation.cc:407] Connection header received
0927 22:14:50.516233 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.516237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.516330 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.516412 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.518416 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.519169 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.520800 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.521199 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.521291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.524709 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.524714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.524717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.524792 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.528074 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.528723 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.528727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.528729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.528905 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.529266 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.529270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.529272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.529546 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.529658 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.530393 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.530606 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:50.531857 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.512456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34223 (local address 127.23.173.67:57385)
0927 22:14:50.513270 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.513315 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.515935 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.516663 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.516675 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.517444 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:50.518237 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.518273 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.519347 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.519351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.520649 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.520661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.521400 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.521405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.521599 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.522362 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:50.522390 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:50.524512 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:50.528203 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:50.528210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:50.528221 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:50.528619 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:50.528999 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:50.529002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:50.529004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:50.529171 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:50.529654 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:50.529660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:50.530018 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.530244 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.530668 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":344,"threads_started":1}
I20260927 22:14:50.533248 25277 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34223
I20260927 22:14:50.533535 25277 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:50.534274 25277 heartbeater.cc:511] Master 127.23.173.126:34223 requested a full tablet report, sending...
I20260927 22:14:50.535615 24820 ts_manager.cc:194] Registered new tserver with Master: fcfc9c3d40b04de0bed6fac38fd9b820 (127.23.173.67:41543)
I20260927 22:14:50.536542 24820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:57385
I20260927 22:14:50.539548 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:14:50.547920 25010 heartbeater.cc:503] Master 127.23.173.126:34223 was elected leader, sending a full tablet report...
I20260927 22:14:50.559142 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.545586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.23.173.126:34223)
0927 22:14:50.545807 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.545814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.546264 (+ 450us) server_negotiation.cc:407] Connection header received
0927 22:14:50.546441 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.546444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.546501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.546573 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.548341 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.549165 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.551109 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.551528 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.551625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.554084 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.554089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.554092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.554122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.556628 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.557274 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.557277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.557279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.557415 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.557744 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.557747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.557749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.558115 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.558284 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.558642 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.558804 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:14:50.565515 24820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:14:50.573274 24820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:50.600940 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.585907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:41543 (local address 127.0.0.1:56702)
0927 22:14:50.586709 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.586757 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.586975 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.591570 (+ 4595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.591581 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.591641 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:50.592200 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.592214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.595219 (+ 3005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.595228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.597932 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.597948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.599874 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.599880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.600080 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.600155 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.600266 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.600501 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":431,"threads_started":1}
I20260927 22:14:50.602000 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.586507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.23.173.67:41543)
0927 22:14:50.590690 (+ 4183us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.590703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.590810 (+ 107us) server_negotiation.cc:407] Connection header received
0927 22:14:50.590935 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.590946 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.591260 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.592467 (+ 1207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:50.592841 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.595022 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.598203 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.599634 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.600697 (+ 1063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.601049 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.601151 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.601398 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3939,"thread_start_us":4979,"threads_started":1}
I20260927 22:14:50.605054 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.585948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38465 (local address 127.0.0.1:33764)
0927 22:14:50.587890 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.587911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.588039 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.597616 (+ 9577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.597621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.597647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:50.598117 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.598130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.600186 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.600190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.602364 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.602383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.604304 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.604312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.604491 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.604527 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.604593 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.604725 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"threads_started":1}
I20260927 22:14:50.606482 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.586078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.23.173.65:38465)
0927 22:14:50.596839 (+ 10761us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.596848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.596876 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:14:50.596968 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.596976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.597212 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.598344 (+ 1132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:50.598552 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.600045 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.602590 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.604050 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.605706 (+ 1656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.606052 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.606119 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.606250 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10575,"thread_start_us":11256,"threads_started":1}
I20260927 22:14:50.606869 25212 tablet_service.cc:1467] Processing CreateTablet for tablet c23e29c5be134ef2953c0f73dc9e1c26 (DEFAULT_TABLE table=test-table [id=4ee0f399962b48c7971182da521c147a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:50.608489 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23e29c5be134ef2953c0f73dc9e1c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:50.610478 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.587244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34717 (local address 127.0.0.1:54912)
0927 22:14:50.601209 (+ 13965us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.601228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.601327 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.603588 (+ 2261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.603592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.603616 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:50.603776 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.603790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.605983 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.605989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.607905 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.607923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.609707 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.609713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.609887 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.609921 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.610020 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.610170 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13866,"thread_start_us":686,"threads_started":1}
I20260927 22:14:50.610077 24945 tablet_service.cc:1467] Processing CreateTablet for tablet c23e29c5be134ef2953c0f73dc9e1c26 (DEFAULT_TABLE table=test-table [id=4ee0f399962b48c7971182da521c147a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:50.611660 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.587800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.23.173.66:34717)
0927 22:14:50.600285 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.600298 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.602003 (+ 1705us) server_negotiation.cc:407] Connection header received
0927 22:14:50.602111 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.602119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.603376 (+ 1257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.603948 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:50.604166 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.605709 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.608089 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.609514 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.610906 (+ 1392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.611113 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.611182 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.611379 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12252,"thread_start_us":12517,"threads_started":1}
I20260927 22:14:50.611733 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23e29c5be134ef2953c0f73dc9e1c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:50.615594 25079 tablet_service.cc:1467] Processing CreateTablet for tablet c23e29c5be134ef2953c0f73dc9e1c26 (DEFAULT_TABLE table=test-table [id=4ee0f399962b48c7971182da521c147a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:50.617841 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23e29c5be134ef2953c0f73dc9e1c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:50.622833 25295 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Bootstrap starting.
I20260927 22:14:50.624899 25296 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Bootstrap starting.
I20260927 22:14:50.626641 25295 tablet_bootstrap.cc:654] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:50.628096 25295 log.cc:824] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:50.628635 25296 tablet_bootstrap.cc:654] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:50.629817 25296 log.cc:824] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:50.631217 25295 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: No bootstrap required, opened a new log
I20260927 22:14:50.631502 25295 ts_tablet_manager.cc:1397] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:14:50.632918 25296 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: No bootstrap required, opened a new log
I20260927 22:14:50.633177 25296 ts_tablet_manager.cc:1397] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:14:50.633824 25299 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Bootstrap starting.
I20260927 22:14:50.637580 25299 tablet_bootstrap.cc:654] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:50.638757 25299 log.cc:824] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:50.641608 25299 tablet_bootstrap.cc:492] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: No bootstrap required, opened a new log
I20260927 22:14:50.641956 25299 ts_tablet_manager.cc:1397] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:14:50.642513 25295 raft_consensus.cc:348] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.642968 25295 raft_consensus.cc:374] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:50.643121 25295 raft_consensus.cc:729] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bebf4873d034dbb9dc3c06969138f07, State: Initialized, Role: FOLLOWER
I20260927 22:14:50.643745 25295 consensus_queue.cc:260] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.643985 25296 raft_consensus.cc:348] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.644487 25296 raft_consensus.cc:374] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:50.644629 25296 raft_consensus.cc:729] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc9c3d40b04de0bed6fac38fd9b820, State: Initialized, Role: FOLLOWER
I20260927 22:14:50.645678 25295 ts_tablet_manager.cc:1428] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 22:14:50.646782 25296 consensus_queue.cc:260] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.647141 25302 raft_consensus.cc:482] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:50.647501 25302 raft_consensus.cc:504] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.653043 25299 raft_consensus.cc:348] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.653497 25299 raft_consensus.cc:374] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:50.653672 25299 raft_consensus.cc:729] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb4fbd0ff694dc5970aaf8306100289, State: Initialized, Role: FOLLOWER
I20260927 22:14:50.654556 25299 consensus_queue.cc:260] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.656054 25277 heartbeater.cc:503] Master 127.23.173.126:34223 was elected leader, sending a full tablet report...
I20260927 22:14:50.656903 25296 ts_tablet_manager.cc:1428] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Time spent starting tablet: real 0.024s user 0.012s sys 0.008s
I20260927 22:14:50.658425 25144 heartbeater.cc:503] Master 127.23.173.126:34223 was elected leader, sending a full tablet report...
I20260927 22:14:50.659233 25299 ts_tablet_manager.cc:1428] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260927 22:14:50.659829 25302 leader_election.cc:288] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bebf4873d034dbb9dc3c06969138f07 (127.23.173.65:38465), cfb4fbd0ff694dc5970aaf8306100289 (127.23.173.66:34717)
I20260927 22:14:50.660247 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.650931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:35891 (local address 127.23.173.65:38465)
0927 22:14:50.651158 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.651166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.651189 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:14:50.651282 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.651290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.651368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.651458 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:50.652047 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.653427 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.657460 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.658823 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.658975 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.659048 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.659899 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.660044 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:14:50.660601 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.650552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38465 (local address 127.23.173.67:35891)
0927 22:14:50.650828 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.650852 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.650970 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.651625 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.651630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.651659 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:50.651841 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.651853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.653591 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.653600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.656151 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.656175 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.659159 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.659166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.659382 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.659417 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.659511 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.659643 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 22:14:50.661552 24965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23e29c5be134ef2953c0f73dc9e1c26" candidate_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bebf4873d034dbb9dc3c06969138f07" is_pre_election: true
I20260927 22:14:50.662108 24965 raft_consensus.cc:2454] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfc9c3d40b04de0bed6fac38fd9b820 in term 0.
I20260927 22:14:50.662938 25166 leader_election.cc:302] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bebf4873d034dbb9dc3c06969138f07, fcfc9c3d40b04de0bed6fac38fd9b820; no voters:
I20260927 22:14:50.663350 25302 raft_consensus.cc:2781] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:50.663566 25302 raft_consensus.cc:482] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:50.663713 25302 raft_consensus.cc:3037] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:50.664809 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.654203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34717 (local address 127.23.173.67:41933)
0927 22:14:50.657493 (+ 3290us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:50.657516 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:50.657646 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:50.658195 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:50.658200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:50.658230 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:50.658604 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.658621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.660492 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.660499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.662588 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:50.662605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.664211 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.664215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:50.664364 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.664387 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:14:50.664461 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 22:14:50.664552 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3172,"thread_start_us":5296,"threads_started":1}
I20260927 22:14:50.664911 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.654828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:41933 (local address 127.23.173.66:34717)
0927 22:14:50.655071 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.655080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.657612 (+ 2532us) server_negotiation.cc:407] Connection header received
0927 22:14:50.657786 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.657795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.657899 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.658035 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:50.658802 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.659867 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.662795 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.664035 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.664204 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.664286 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.664506 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.664643 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:14:50.665892 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23e29c5be134ef2953c0f73dc9e1c26" candidate_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb4fbd0ff694dc5970aaf8306100289" is_pre_election: true
I20260927 22:14:50.666355 25099 raft_consensus.cc:2454] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfc9c3d40b04de0bed6fac38fd9b820 in term 0.
I20260927 22:14:50.666724 25302 raft_consensus.cc:504] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.667681 25302 leader_election.cc:288] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [CANDIDATE]: Term 1 election: Requested vote from peers 5bebf4873d034dbb9dc3c06969138f07 (127.23.173.65:38465), cfb4fbd0ff694dc5970aaf8306100289 (127.23.173.66:34717)
I20260927 22:14:50.667924 24965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23e29c5be134ef2953c0f73dc9e1c26" candidate_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bebf4873d034dbb9dc3c06969138f07"
I20260927 22:14:50.668118 24965 raft_consensus.cc:3037] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:50.668391 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23e29c5be134ef2953c0f73dc9e1c26" candidate_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb4fbd0ff694dc5970aaf8306100289"
I20260927 22:14:50.668573 25099 raft_consensus.cc:3037] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:50.671451 24965 raft_consensus.cc:2454] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfc9c3d40b04de0bed6fac38fd9b820 in term 1.
I20260927 22:14:50.671686 25099 raft_consensus.cc:2454] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfc9c3d40b04de0bed6fac38fd9b820 in term 1.
I20260927 22:14:50.671932 25166 leader_election.cc:302] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bebf4873d034dbb9dc3c06969138f07, fcfc9c3d40b04de0bed6fac38fd9b820; no voters:
I20260927 22:14:50.672195 25302 raft_consensus.cc:2781] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:50.672379 25302 raft_consensus.cc:686] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [term 1 LEADER]: Becoming Leader. State: Replica: fcfc9c3d40b04de0bed6fac38fd9b820, State: Running, Role: LEADER
I20260927 22:14:50.672796 25302 consensus_queue.cc:237] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } }
I20260927 22:14:50.676752 24819 catalog_manager.cc:5495] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcfc9c3d40b04de0bed6fac38fd9b820 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bebf4873d034dbb9dc3c06969138f07" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfc9c3d40b04de0bed6fac38fd9b820" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 41543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb4fbd0ff694dc5970aaf8306100289" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34717 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:50.707559 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.689644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.23.173.65:38465)
0927 22:14:50.689818 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.689823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.690384 (+ 561us) server_negotiation.cc:407] Connection header received
0927 22:14:50.690592 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.690596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.690667 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.690767 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.692263 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.693021 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.694307 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.694739 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.695150 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.700536 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.700562 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.700586 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.700641 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.704882 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.705685 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.705694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.705696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.705840 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.706200 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.706203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.706205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.706582 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.706820 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.707020 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.707180 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:50.754122 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.740173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.23.173.65:38465)
0927 22:14:50.740408 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.740414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.740594 (+ 180us) server_negotiation.cc:407] Connection header received
0927 22:14:50.740866 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.740870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.740937 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.741044 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.742549 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.743282 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.744614 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.745086 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.745219 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.748907 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.748912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.748915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.748947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.751661 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.752201 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.752206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.752208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.752358 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.752807 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.752828 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.752831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.753075 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.753260 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.753565 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.753727 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 22:14:50.758077 25145 tablet.cc:2369] T c23e29c5be134ef2953c0f73dc9e1c26 P cfb4fbd0ff694dc5970aaf8306100289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:14:50.761602 25278 tablet.cc:2369] T c23e29c5be134ef2953c0f73dc9e1c26 P fcfc9c3d40b04de0bed6fac38fd9b820: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:14:50.765538 25011 tablet.cc:2369] T c23e29c5be134ef2953c0f73dc9e1c26 P 5bebf4873d034dbb9dc3c06969138f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:50.800112 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.786085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.23.173.65:38465)
0927 22:14:50.786280 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.786288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.786625 (+ 337us) server_negotiation.cc:407] Connection header received
0927 22:14:50.786834 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.786839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.786904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.787010 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.788366 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.789056 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.790355 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.790791 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.790932 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.794882 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.794887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.794890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.794926 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.797408 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.798142 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.798146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.798149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.798277 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.798652 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.798662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.798664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.798962 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.799214 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.799514 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.799769 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:14:50.801868 24925 tablet_service.cc:2908] Rejecting scan request for tablet c23e29c5be134ef2953c0f73dc9e1c26: Uninitialized: clean time has not yet been initialized
I20260927 22:14:50.805034 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24882
I20260927 22:14:50.817111 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.806408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.23.173.65:38465)
0927 22:14:50.806619 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.806624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.806822 (+ 198us) server_negotiation.cc:407] Connection header received
0927 22:14:50.806944 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.806947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.807012 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.807089 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.808187 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.808842 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.809932 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.810371 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.810473 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.812519 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.812523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.812526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.812552 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.814954 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.815528 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.815532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.815534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.815642 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.815947 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.815951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.815953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.816204 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.816407 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.816593 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.816784 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:14:50.817885 25005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:50.907644 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24882
I20260927 22:14:50.930467 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25016
Sep 27 22:14:50 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:14:50.946472 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:50.931304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.23.173.66:34717)
0927 22:14:50.931474 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:50.931479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:50.931497 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:14:50.931682 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:50.931686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:50.931795 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:50.931880 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:50.933366 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.934266 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.936091 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:50.936585 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:50.936703 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:50.941191 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:50.941205 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:50.941229 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:50.941268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:50.943984 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.944589 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.944596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.944598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.944721 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:50.945045 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:50.945048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:50.945050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:50.945405 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:50.945722 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:50.945907 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 22:14:50.946125 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:14:50.947106 25139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:51.033138 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25016
I20260927 22:14:51.049806 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25149
Sep 27 22:14:51 dist-test-slave-3wc3 krb5kdc[24775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:14:51.064090 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:51.050667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.23.173.67:41543)
0927 22:14:51.050901 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:51.050909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:51.050932 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:14:51.051041 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:51.051047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:51.051158 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:51.051271 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:51.052538 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:51.053342 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:51.054803 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:51.055219 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:51.055355 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:51.059241 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:51.059255 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:51.059272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:51.059304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:51.061865 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:51.062496 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:51.062503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:51.062505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:51.062666 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:51.062972 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:51.062975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:51.062977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:51.063302 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:51.063522 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:51.063678 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:14:51.063858 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:14:51.064719 25272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:51.151680 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25149
I20260927 22:14:51.167743 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 24788
I20260927 22:14:51.168895 24850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:51.275683 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 24788
2026-09-27T22:14:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4072 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:51 dist-test-slave-3wc3 krb5kdc[25324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:51 dist-test-slave-3wc3 krb5kdc[25324](info): set up 2 sockets
Sep 27 22:14:51 dist-test-slave-3wc3 krb5kdc[25324](info): commencing operation
krb5kdc: starting...
W20260927 22:14:52.362257 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:14:52 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:14:52.541092 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42363
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:52.856782 25337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:52.857107 25337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:52.857270 25337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:52.864446 25337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:52.864606 25337 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:14:52.864696 25337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:52.864804 25337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:52.864890 25337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:52.864979 25337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:52.875269 25337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:52.876606 25337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:52.877948 25337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:52.889940 25343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:52.890003 25342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:52.890026 25345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:52.890661 25337 server_base.cc:1034] running on GCE node
I20260927 22:14:52.891383 25337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:52.892625 25337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:52.893863 25337 hybrid_clock.cc:648] HybridClock initialized: now 1790547292893794 us; error 42 us; skew 500 ppm
I20260927 22:14:52.894200 25337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:52 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547292, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:52.897284 25337 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:14:52.898692 25337 webserver.cc:460] Webserver started at http://127.23.173.126:45819/ using document root <none> and password file <none>
I20260927 22:14:52.899317 25337 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:52.899415 25337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:52.899667 25337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:52.902522 25337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "0a7bba45b35d4d7d8e051e05312bfadf"
format_stamp: "Formatted at 2026-09-27 22:14:52 on dist-test-slave-3wc3"
server_key: "94cffbd548d72bbc3011fc4edc272dac"
server_key_iv: "99e3decc548f3b52200d1b41c504cb9a"
server_key_version: "encryptionkey@0"
I20260927 22:14:52.903227 25337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "0a7bba45b35d4d7d8e051e05312bfadf"
format_stamp: "Formatted at 2026-09-27 22:14:52 on dist-test-slave-3wc3"
server_key: "94cffbd548d72bbc3011fc4edc272dac"
server_key_iv: "99e3decc548f3b52200d1b41c504cb9a"
server_key_version: "encryptionkey@0"
I20260927 22:14:52.908118 25337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:14:52.911383 25352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:52.912758 25337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:52.912943 25337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "0a7bba45b35d4d7d8e051e05312bfadf"
format_stamp: "Formatted at 2026-09-27 22:14:52 on dist-test-slave-3wc3"
server_key: "94cffbd548d72bbc3011fc4edc272dac"
server_key_iv: "99e3decc548f3b52200d1b41c504cb9a"
server_key_version: "encryptionkey@0"
I20260927 22:14:52.913143 25337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:52.942938 25337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:52.946898 25337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:52.947160 25337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:52.968511 25337 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42363
I20260927 22:14:52.968550 25413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42363 every 8 connection(s)
I20260927 22:14:52.969801 25337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:52.973389 25414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:52.976799 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25337
I20260927 22:14:52.977023 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:14:52.977671 24245 external_mini_cluster.cc:1428] Setting key bee5d1ff62fd01961a3bd664f60d0786
I20260927 22:14:52.986438 25414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Bootstrap starting.
I20260927 22:14:52.990476 25414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:52.991775 25414 log.cc:824] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:14:52 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:52.994582 25414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: No bootstrap required, opened a new log
I20260927 22:14:53.000526 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:52.979537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.23.173.126:42363)
0927 22:14:52.980268 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:52.980278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:52.980317 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:14:52.981890 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:52.981917 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:52.982448 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:52.982939 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:52.984707 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:52.986090 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:52.987794 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:52.988635 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:52.988817 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:52.993398 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:52.993407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:52.993421 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:52.993456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:52.996539 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:52.997229 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:52.997241 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:52.997247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:52.997412 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:52.997767 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:52.997772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:52.997783 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:52.998356 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:52.998676 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:52.999098 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 22:14:52.999489 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":340,"threads_started":1}
I20260927 22:14:53.002277 25414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } }
I20260927 22:14:53.002627 25414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:53.002743 25414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7bba45b35d4d7d8e051e05312bfadf, State: Initialized, Role: FOLLOWER
I20260927 22:14:53.003393 25414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } }
I20260927 22:14:53.003648 25414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:53.003783 25414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:53.003928 25414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:53.006008 25414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } }
I20260927 22:14:53.006485 25414 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7bba45b35d4d7d8e051e05312bfadf; no voters:
I20260927 22:14:53.007110 25414 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:53.007227 25419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:53.008205 25419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [term 1 LEADER]: Becoming Leader. State: Replica: 0a7bba45b35d4d7d8e051e05312bfadf, State: Running, Role: LEADER
I20260927 22:14:53.008685 25419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } }
I20260927 22:14:53.009375 25414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:53.011554 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7bba45b35d4d7d8e051e05312bfadf. Latest consensus state: current_term: 1 leader_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } } }
I20260927 22:14:53.012019 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:53.012919 25420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7bba45b35d4d7d8e051e05312bfadf" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42363 } } }
I20260927 22:14:53.013233 25420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:53.013723 25425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:53.018792 25425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:53.027068 25425 catalog_manager.cc:1347] Generated new cluster ID: ec581dcc8d9e48b4922d963fab2c1e29
I20260927 22:14:53.027184 25425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:53.050143 25425 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:53.051501 25425 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:53.057854 25425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Generated new TSK 0
I20260927 22:14:53.058691 25425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:14:53.103315 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42363
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:53.419586 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:53.419872 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:53.420029 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:53.427482 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:53.427603 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:53.427762 25440 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:53.427906 25440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:14:53.437785 25440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:53.439196 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:53.440480 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:53.453677 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:53.453744 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:53.453693 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:53.454718 25440 server_base.cc:1034] running on GCE node
I20260927 22:14:53.455283 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:53.456146 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:53.457371 25440 hybrid_clock.cc:648] HybridClock initialized: now 1790547293457303 us; error 40 us; skew 500 ppm
I20260927 22:14:53.457684 25440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:53 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:53.461055 25440 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:14:53.462579 25440 webserver.cc:460] Webserver started at http://127.23.173.65:34731/ using document root <none> and password file <none>
I20260927 22:14:53.463223 25440 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:53.463307 25440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:53.463591 25440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:53.466432 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "ad901bd6aa994c968aea6a808d1c6a89"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "90f0ab14b42ea813b6b5bac26c578564"
server_key_iv: "c29606c35e0d21ce715d51fc221777a1"
server_key_version: "encryptionkey@0"
I20260927 22:14:53.467144 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "ad901bd6aa994c968aea6a808d1c6a89"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "90f0ab14b42ea813b6b5bac26c578564"
server_key_iv: "c29606c35e0d21ce715d51fc221777a1"
server_key_version: "encryptionkey@0"
I20260927 22:14:53.472177 25440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:14:53.475431 25455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:53.476715 25440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:14:53.476871 25440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "ad901bd6aa994c968aea6a808d1c6a89"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "90f0ab14b42ea813b6b5bac26c578564"
server_key_iv: "c29606c35e0d21ce715d51fc221777a1"
server_key_version: "encryptionkey@0"
I20260927 22:14:53.477062 25440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:53.504225 25440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:53.508306 25440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:53.508582 25440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:53.511353 25440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:53.511466 25440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:53.511606 25440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:53.511685 25440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:53 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:53.536121 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:53.511885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:42529 (local address 127.23.173.126:42363)
0927 22:14:53.512589 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:53.512595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:53.513847 (+ 1252us) server_negotiation.cc:407] Connection header received
0927 22:14:53.517229 (+ 3382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:53.517236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:53.517319 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:53.517431 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:53.519801 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.520834 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.523235 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.523839 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.523982 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:53.529219 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:53.529224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:53.529227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:53.529259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:53.532876 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:53.533612 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:53.533620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:53.533626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:53.533805 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:53.534397 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:53.534402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:53.534410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:53.534690 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:53.534862 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:53.535534 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 22:14:53.535766 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":439,"threads_started":1}
I20260927 22:14:53.537088 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:53.512641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.65:42529)
0927 22:14:53.513652 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:53.513713 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:53.516714 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:53.517785 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:53.517793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:53.518584 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:53.519592 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:53.519624 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.520982 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.520988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:53.523063 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:53.523087 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.524090 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.524097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:53.524432 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:53.525251 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:53.525298 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:53.528998 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:53.533031 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:53.533040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:53.533057 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:53.533479 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:53.533944 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:53.533948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:53.533950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:53.534259 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:53.534830 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:53.534840 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:53.535176 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 22:14:53.535440 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 22:14:53.535939 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":423,"threads_started":1}
I20260927 22:14:53.557777 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:53.542564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:49775 (local address 127.23.173.126:42363)
0927 22:14:53.542787 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:53.542796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:53.543370 (+ 574us) server_negotiation.cc:407] Connection header received
0927 22:14:53.543558 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:53.543561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:53.543618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:53.543695 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:53.545149 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.545829 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.548304 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.548800 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.548906 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:53.551766 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:53.551770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:53.551774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:53.551804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:53.555028 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:53.555720 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:53.555727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:53.555730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:53.555878 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:53.556261 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:53.556268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:53.556271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:53.556562 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:53.556713 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:53.557153 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 22:14:53.557385 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:14:53.557840 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:53.542529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.65:49775)
0927 22:14:53.543253 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:53.543278 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:53.543439 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:53.543753 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:53.543760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:53.544540 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:53.544997 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:53.545013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.546007 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.546015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:53.548128 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:53.548147 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:53.549014 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:53.549022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:53.549279 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:53.550142 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:53.550176 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:53.551577 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:53.555172 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:53.555178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:53.555182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:53.555599 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:53.555999 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:53.556004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:53.556007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:53.556142 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:53.556678 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:53.556683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:53.556893 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 22:14:53.557036 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:14:53.557383 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":371,"threads_started":1}
I20260927 22:14:53.571923 25440 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:45937
I20260927 22:14:53.571972 25575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:45937 every 8 connection(s)
I20260927 22:14:53.573134 25440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:14:53.575204 25576 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42363
I20260927 22:14:53.575440 25576 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:53.576081 25576 heartbeater.cc:511] Master 127.23.173.126:42363 requested a full tablet report, sending...
I20260927 22:14:53.578042 25369 ts_manager.cc:194] Registered new tserver with Master: ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937)
I20260927 22:14:53.579501 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:42529
I20260927 22:14:53.580628 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25440
I20260927 22:14:53.580847 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:14:53.581653 24245 external_mini_cluster.cc:1428] Setting key bada813e9e0482399c9f90e8467daf4e
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:14:53.619158 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42363
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:53.941247 25580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:53.941499 25580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:53.941592 25580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:53.948680 25580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:53.948787 25580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:53.948870 25580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:53.948949 25580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:14:53.958280 25580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:53.959440 25580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:53.960696 25580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:53.974011 25585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:53.974072 25588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:53.974499 25580 server_base.cc:1034] running on GCE node
W20260927 22:14:53.977839 25586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:53.978531 25580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:53.979409 25580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:53.980638 25580 hybrid_clock.cc:648] HybridClock initialized: now 1790547293980569 us; error 44 us; skew 500 ppm
I20260927 22:14:53.980944 25580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:53 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:53.987296 25580 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:14:53.988729 25580 webserver.cc:460] Webserver started at http://127.23.173.66:33649/ using document root <none> and password file <none>
I20260927 22:14:53.989424 25580 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:53.989511 25580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:53.989827 25580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:53.992722 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "5e692b1469fb42768a17e4852c05914a"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "30241f29208894b8d93a4d8028bc50af"
server_key_iv: "f5a9be45d6c611a01a78489055c2b644"
server_key_version: "encryptionkey@0"
I20260927 22:14:53.993465 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "5e692b1469fb42768a17e4852c05914a"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "30241f29208894b8d93a4d8028bc50af"
server_key_iv: "f5a9be45d6c611a01a78489055c2b644"
server_key_version: "encryptionkey@0"
I20260927 22:14:53.998392 25580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:14:54.001639 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:54.002957 25580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:54.003115 25580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "5e692b1469fb42768a17e4852c05914a"
format_stamp: "Formatted at 2026-09-27 22:14:53 on dist-test-slave-3wc3"
server_key: "30241f29208894b8d93a4d8028bc50af"
server_key_iv: "f5a9be45d6c611a01a78489055c2b644"
server_key_version: "encryptionkey@0"
I20260927 22:14:54.003306 25580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:54.050352 25580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:54.056342 25580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:54.056646 25580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:54.060879 25580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:54.061005 25580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:54.061121 25580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:54.061178 25580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:54 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:54.081449 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.060737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:41631 (local address 127.23.173.126:42363)
0927 22:14:54.061186 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.061193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.062442 (+ 1249us) server_negotiation.cc:407] Connection header received
0927 22:14:54.064536 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.064541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.064630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.064754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:54.066952 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.068009 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.069607 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.070135 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.070274 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.075221 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:54.075226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:54.075229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:54.075262 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:54.078351 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.079069 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.079075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.079078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.079204 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.079618 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.079622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.079626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.079905 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:54.080059 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.080815 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.081052 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":239,"threads_started":1}
I20260927 22:14:54.082489 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.061385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.66:41631)
0927 22:14:54.062276 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.062328 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.064262 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.064991 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.064999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.065781 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:54.066767 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.066803 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.068159 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.068164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.069472 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.069483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.070409 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.070416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.070799 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.071598 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:54.071638 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:54.075013 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:54.078482 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.078491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.078512 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.078940 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.079308 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.079316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.079319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.079497 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.080048 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:54.080056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:54.080395 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.080707 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.081249 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":365,"threads_started":1}
I20260927 22:14:54.108784 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.092703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:46559 (local address 127.23.173.126:42363)
0927 22:14:54.092935 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.092943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.093633 (+ 690us) server_negotiation.cc:407] Connection header received
0927 22:14:54.093897 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.093907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.094011 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.094107 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:54.095834 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.096558 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.098962 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.099338 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.099441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.102149 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:54.102157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:54.102163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:54.102205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:54.105549 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.106629 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.106636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.106639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.106759 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.107218 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.107221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.107223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.107685 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:54.107848 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.108355 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.108549 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:14:54.109354 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.092785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.66:46559)
0927 22:14:54.093511 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.093537 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.093696 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.094167 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.094176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.095049 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:54.095660 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.095680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.096696 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.096705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.098799 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.098817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.099531 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.099540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.099814 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.100499 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:54.100530 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:54.101851 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:54.105718 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.105730 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.105735 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.106484 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.106868 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.106875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.106881 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.107083 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.107822 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:54.107830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:54.108096 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.108275 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.108695 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260927 22:14:54.127916 25580 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34353
I20260927 22:14:54.127964 25715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34353 every 8 connection(s)
I20260927 22:14:54.129483 25580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:14:54.132794 25716 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42363
I20260927 22:14:54.133033 25716 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:54.133661 25716 heartbeater.cc:511] Master 127.23.173.126:42363 requested a full tablet report, sending...
I20260927 22:14:54.135059 25369 ts_manager.cc:194] Registered new tserver with Master: 5e692b1469fb42768a17e4852c05914a (127.23.173.66:34353)
I20260927 22:14:54.135833 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:41631
I20260927 22:14:54.136485 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25580
I20260927 22:14:54.136684 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:14:54.137112 24245 external_mini_cluster.cc:1428] Setting key 1a0e35030aa2be92f31067aa02967a85
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:14:54.176928 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42363
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:54.500298 25720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:54.500602 25720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:54.500742 25720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:54.507915 25720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:54.508013 25720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:54.508095 25720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:54.508173 25720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:14:54.517928 25720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:54.519086 25720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:54.520327 25720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:54.533093 25726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:54.533142 25728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:54.533324 25725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:54.533880 25720 server_base.cc:1034] running on GCE node
I20260927 22:14:54.534515 25720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:54.535341 25720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:54.536547 25720 hybrid_clock.cc:648] HybridClock initialized: now 1790547294536469 us; error 54 us; skew 500 ppm
I20260927 22:14:54.536798 25720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:54 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:54.539870 25720 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:14:54.541249 25720 webserver.cc:460] Webserver started at http://127.23.173.67:36885/ using document root <none> and password file <none>
I20260927 22:14:54.541899 25720 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:54.542069 25720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:54.542340 25720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:54.545094 25720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
format_stamp: "Formatted at 2026-09-27 22:14:54 on dist-test-slave-3wc3"
server_key: "74fb3bca509f64d0265920b92558fbaf"
server_key_iv: "e6c6de30a18bedacf3ad342de85a5500"
server_key_version: "encryptionkey@0"
I20260927 22:14:54.545841 25720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
format_stamp: "Formatted at 2026-09-27 22:14:54 on dist-test-slave-3wc3"
server_key: "74fb3bca509f64d0265920b92558fbaf"
server_key_iv: "e6c6de30a18bedacf3ad342de85a5500"
server_key_version: "encryptionkey@0"
I20260927 22:14:54.550768 25720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:54.554087 25735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:54.555420 25720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:54.555559 25720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
format_stamp: "Formatted at 2026-09-27 22:14:54 on dist-test-slave-3wc3"
server_key: "74fb3bca509f64d0265920b92558fbaf"
server_key_iv: "e6c6de30a18bedacf3ad342de85a5500"
server_key_version: "encryptionkey@0"
I20260927 22:14:54.555742 25720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:54.582222 25720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:54.582744 25576 heartbeater.cc:503] Master 127.23.173.126:42363 was elected leader, sending a full tablet report...
I20260927 22:14:54.586971 25720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:54.587244 25720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:54.590040 25720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:54.590133 25720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:54.590263 25720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:54.590360 25720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:54 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:54.613899 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.590624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:36627 (local address 127.23.173.126:42363)
0927 22:14:54.590829 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.590836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.592156 (+ 1320us) server_negotiation.cc:407] Connection header received
0927 22:14:54.594185 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.594190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.594275 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.594388 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:54.596628 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.597736 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.600475 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.600919 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.601087 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.606277 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:54.606285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:54.606292 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:54.606341 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:54.610256 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.611239 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.611248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.611254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.611449 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.611936 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.611944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.611950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.612411 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:54.612591 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.613304 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.613520 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:14:54.615114 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.591201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.67:36627)
0927 22:14:54.592012 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.592046 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.593876 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.594645 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.594659 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.595486 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:54.596398 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.596419 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.597917 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.597927 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.600329 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.600340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.601385 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.601391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.601637 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.602467 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:54.602506 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:54.606012 (+ 3506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:54.610411 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.610427 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.610447 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.611095 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.611566 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.611569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.611571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.611757 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.612547 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:54.612552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:54.612916 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.613190 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.613686 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":347,"threads_started":1}
I20260927 22:14:54.636914 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.621433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:54809 (local address 127.23.173.126:42363)
0927 22:14:54.621585 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.621589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.622613 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 22:14:54.622817 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.622822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.622891 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.622987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:54.624449 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.625559 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.627662 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.628384 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.628527 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.630319 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:54.630327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:54.630334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:54.630381 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:54.633690 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.634577 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.634583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.634587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.634822 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.635330 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.635333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.635335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.635581 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:54.635746 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.636342 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.636551 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:14:54.637012 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.621530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.23.173.67:54809)
0927 22:14:54.622516 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.622532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.622677 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.623015 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.623019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.623794 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:54.624306 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.624324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.625688 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.625693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.627499 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.627513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.628590 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.628598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.628771 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.629376 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:54.629392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:54.630141 (+ 749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:54.633809 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.633814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.633818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.634454 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.634970 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:54.634978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:54.634985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:54.635187 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:54.635872 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:54.635878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:54.636085 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.636235 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.636543 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":608,"threads_started":1}
I20260927 22:14:54.643849 25720 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:39149
I20260927 22:14:54.643893 25854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:39149 every 8 connection(s)
I20260927 22:14:54.645013 25720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:14:54.646878 25855 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42363
I20260927 22:14:54.647094 25855 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:54.647706 25855 heartbeater.cc:511] Master 127.23.173.126:42363 requested a full tablet report, sending...
I20260927 22:14:54.648996 25369 ts_manager.cc:194] Registered new tserver with Master: c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149)
I20260927 22:14:54.649752 25369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:36627
I20260927 22:14:54.654057 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25720
I20260927 22:14:54.654206 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:14:54.654587 24245 external_mini_cluster.cc:1428] Setting key 5ed111e07ab54efa0c730a930f72d185
I20260927 22:14:54.657276 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:14:54 dist-test-slave-3wc3 krb5kdc[25324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:14:54 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:54.701030 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.687731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.23.173.126:42363)
0927 22:14:54.687889 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.687893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.688262 (+ 369us) server_negotiation.cc:407] Connection header received
0927 22:14:54.688433 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.688437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.688500 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.688578 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:54.689877 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.690678 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.692044 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.692437 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.692530 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.696345 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:54.696350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:54.696353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:54.696386 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:54.698904 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.699468 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.699472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.699474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.699588 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:54.699886 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:54.699889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:54.699891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:54.700162 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:54.700344 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.700547 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.700699 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:14:54.704839 25369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:14:54.711901 25369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:54.736822 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.725127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.23.173.65:45937)
0927 22:14:54.726671 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.726683 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.728656 (+ 1973us) server_negotiation.cc:407] Connection header received
0927 22:14:54.728791 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.728803 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.729151 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.729373 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:54.730296 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.731909 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.734224 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.735471 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.735604 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.735839 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.736348 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.736512 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":1169,"threads_started":1}
I20260927 22:14:54.736960 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.724483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:45937 (local address 127.0.0.1:39484)
0927 22:14:54.728366 (+ 3883us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.728387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.728515 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.729597 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.729603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.729630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:54.730091 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.730105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.732128 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.732134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.734072 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.734087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.735963 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.735969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.736130 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.736163 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.736295 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.736490 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3708,"mutex_wait_us":43,"thread_start_us":2364,"threads_started":1}
I20260927 22:14:54.738668 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.724224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.23.173.67:39149)
0927 22:14:54.729315 (+ 5091us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.729326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.729359 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:14:54.729474 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.729484 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.729728 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.730734 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:54.731055 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.732437 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.735583 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.737238 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.737383 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.737629 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.738184 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.738340 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4917,"thread_start_us":4747,"threads_started":1}
I20260927 22:14:54.738778 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.725987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34353 (local address 127.0.0.1:49290)
0927 22:14:54.729628 (+ 3641us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.729647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.729791 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.730410 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.730414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.730443 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:54.730848 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.730863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.732631 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.732637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.735381 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.735396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.737862 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.737868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.738053 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.738084 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.738181 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.738363 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3513,"mutex_wait_us":114,"thread_start_us":3363,"threads_started":1}
I20260927 22:14:54.738770 25868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.723946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:39149 (local address 127.0.0.1:46388)
0927 22:14:54.725545 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.725595 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.725759 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.729995 (+ 4236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.730002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.730033 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:54.730489 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.730507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.732561 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.732569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.735392 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.735400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.737772 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.737779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.737982 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.738021 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.738131 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.738277 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":152,"thread_start_us":1196,"threads_started":1}
I20260927 22:14:54.739379 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.726330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.23.173.66:34353)
0927 22:14:54.728717 (+ 2387us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.728730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.729767 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 22:14:54.729993 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.730001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.730219 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.730436 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:54.731048 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.732458 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.735572 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.737575 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.737774 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.737983 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.738880 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.739066 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2174,"thread_start_us":2009,"threads_started":1}
I20260927 22:14:54.741423 25506 tablet_service.cc:1467] Processing CreateTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee (DEFAULT_TABLE table=test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:54.742178 25651 tablet_service.cc:1467] Processing CreateTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee (DEFAULT_TABLE table=test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:54.742443 25785 tablet_service.cc:1467] Processing CreateTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee (DEFAULT_TABLE table=test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:54.742983 25506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90060711d6e46ada3c0109f3d6bf7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:54.743433 25651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90060711d6e46ada3c0109f3d6bf7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:54.743782 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90060711d6e46ada3c0109f3d6bf7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:54.754112 25874 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Bootstrap starting.
I20260927 22:14:54.757329 25875 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Bootstrap starting.
I20260927 22:14:54.758419 25874 tablet_bootstrap.cc:654] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:54.759199 25876 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Bootstrap starting.
I20260927 22:14:54.759963 25874 log.cc:824] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:54.761498 25875 tablet_bootstrap.cc:654] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:54.762423 25876 tablet_bootstrap.cc:654] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:54.762789 25875 log.cc:824] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:54.763257 25874 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: No bootstrap required, opened a new log
I20260927 22:14:54.763468 25876 log.cc:824] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:54.763556 25874 ts_tablet_manager.cc:1397] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:14:54.767001 25875 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: No bootstrap required, opened a new log
I20260927 22:14:54.767342 25875 ts_tablet_manager.cc:1397] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:14:54.769616 25876 tablet_bootstrap.cc:492] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: No bootstrap required, opened a new log
I20260927 22:14:54.769877 25876 ts_tablet_manager.cc:1397] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 22:14:54.771487 25874 raft_consensus.cc:348] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.771855 25874 raft_consensus.cc:374] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:54.771979 25874 raft_consensus.cc:729] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Initialized, Role: FOLLOWER
I20260927 22:14:54.772642 25874 consensus_queue.cc:260] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.773725 25716 heartbeater.cc:503] Master 127.23.173.126:42363 was elected leader, sending a full tablet report...
I20260927 22:14:54.774319 25874 ts_tablet_manager.cc:1428] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 22:14:54.777483 25875 raft_consensus.cc:348] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.778000 25875 raft_consensus.cc:374] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:54.778115 25875 raft_consensus.cc:729] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad901bd6aa994c968aea6a808d1c6a89, State: Initialized, Role: FOLLOWER
I20260927 22:14:54.778703 25875 consensus_queue.cc:260] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.780109 25876 raft_consensus.cc:348] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.780529 25876 raft_consensus.cc:374] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:54.780602 25875 ts_tablet_manager.cc:1428] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 22:14:54.780630 25876 raft_consensus.cc:729] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ed0b1798c349abbf5ea0ab2527e3f2, State: Initialized, Role: FOLLOWER
I20260927 22:14:54.781234 25876 consensus_queue.cc:260] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.782332 25855 heartbeater.cc:503] Master 127.23.173.126:42363 was elected leader, sending a full tablet report...
I20260927 22:14:54.783017 25876 ts_tablet_manager.cc:1428] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
W20260927 22:14:54.825521 25577 tablet.cc:2369] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:14:54.881759 25717 tablet.cc:2369] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:14:54.896593 25856 tablet.cc:2369] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:54.971446 25880 raft_consensus.cc:482] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:54.971774 25880 raft_consensus.cc:504] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.973838 25880 leader_election.cc:288] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149), ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937)
I20260927 22:14:54.981997 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.974600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34421 (local address 127.23.173.65:45937)
0927 22:14:54.974859 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.974868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.975101 (+ 233us) server_negotiation.cc:407] Connection header received
0927 22:14:54.975418 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.975426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.975507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.975565 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:54.976269 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.977148 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.979555 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.980777 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.980928 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.981003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.981567 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.981717 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:14:54.982127 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.974424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:40603 (local address 127.23.173.67:39149)
0927 22:14:54.974625 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:54.974634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:54.975699 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 22:14:54.975891 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:54.975900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:54.975956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:54.976049 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:54.976686 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.977750 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.980050 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.981004 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.981186 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.981261 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:54.981697 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 22:14:54.981838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 22:14:54.982446 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.974119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:45937 (local address 127.23.173.66:34421)
0927 22:14:54.974994 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.975014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.975231 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.975754 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.975759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.975784 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:54.976116 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.976126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.977285 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.977292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.979396 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.979410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.981180 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.981188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.981378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.981413 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.981503 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.981642 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":16,"thread_start_us":1192,"threads_started":1}
I20260927 22:14:54.982949 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:54.974236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:39149 (local address 127.23.173.66:40603)
0927 22:14:54.975601 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:54.975615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:54.975760 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:54.976071 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:54.976075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:54.976101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:54.976523 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.976537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.977885 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.977891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.979909 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:54.979921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:54.981273 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:54.981284 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:54.981522 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:54.981554 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:14:54.982409 (+ 855us) client_negotiation.cc:240] Negotiation successful
0927 22:14:54.982561 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":51,"thread_start_us":645,"threads_started":1}
I20260927 22:14:54.983516 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90060711d6e46ada3c0109f3d6bf7ee" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89" is_pre_election: true
I20260927 22:14:54.984123 25531 raft_consensus.cc:2454] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:54.984010 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90060711d6e46ada3c0109f3d6bf7ee" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" is_pre_election: true
I20260927 22:14:54.984575 25806 raft_consensus.cc:2454] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:54.984975 25596 leader_election.cc:302] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, ad901bd6aa994c968aea6a808d1c6a89; no voters:
I20260927 22:14:54.985448 25880 raft_consensus.cc:2781] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:54.985592 25880 raft_consensus.cc:482] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:54.985683 25880 raft_consensus.cc:3037] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:54.988528 25880 raft_consensus.cc:504] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.989073 25880 leader_election.cc:288] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Requested vote from peers c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149), ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937)
I20260927 22:14:54.989517 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90060711d6e46ada3c0109f3d6bf7ee" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
I20260927 22:14:54.989517 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90060711d6e46ada3c0109f3d6bf7ee" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89"
I20260927 22:14:54.989718 25531 raft_consensus.cc:3037] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:54.989701 25806 raft_consensus.cc:3037] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:54.992136 25806 raft_consensus.cc:2454] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:54.992244 25531 raft_consensus.cc:2454] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:54.992596 25598 leader_election.cc:302] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, c9ed0b1798c349abbf5ea0ab2527e3f2; no voters:
I20260927 22:14:54.992920 25880 raft_consensus.cc:2781] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:54.993328 25880 raft_consensus.cc:686] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 1 LEADER]: Becoming Leader. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Running, Role: LEADER
I20260927 22:14:54.993897 25880 consensus_queue.cc:237] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } }
I20260927 22:14:54.998396 25368 catalog_manager.cc:5495] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e692b1469fb42768a17e4852c05914a (127.23.173.66). New cstate: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } health_report { overall_health: UNKNOWN } } }
I20260927 22:14:55.036128 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.030575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.23.173.66:34353)
0927 22:14:55.030796 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.030802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.030972 (+ 170us) server_negotiation.cc:407] Connection header received
0927 22:14:55.031290 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.031295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.031397 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.031540 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:55.031909 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.032626 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.033950 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.034401 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.034536 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.035043 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:55.035515 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:55.035661 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.035776 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.035855 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:14:55.044036 25531 raft_consensus.cc:1261] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.044036 25806 raft_consensus.cc:1261] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.045056 25885 consensus_queue.cc:1035] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.045446 25880 consensus_queue.cc:1035] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.073264 25369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:14:55.073556 25369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:14:55.076943 25369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Sending DeleteTablet for 3 replicas of tablet f90060711d6e46ada3c0109f3d6bf7ee
I20260927 22:14:55.078143 25506 tablet_service.cc:1514] Processing DeleteTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:39484
I20260927 22:14:55.078236 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:46388
I20260927 22:14:55.078398 25651 tablet_service.cc:1514] Processing DeleteTablet for tablet f90060711d6e46ada3c0109f3d6bf7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:49290
I20260927 22:14:55.080996 25369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:14:55.082769 25369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:55.091205 25898 tablet_replica.cc:330] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: stopping tablet replica
I20260927 22:14:55.091843 25898 raft_consensus.cc:2229] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:14:55.092485 25898 raft_consensus.cc:2258] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.093775 25506 tablet_service.cc:1467] Processing CreateTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 (DEFAULT_TABLE table=test-table [id=a264e4e016964916b0ec2b4709779c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:55.094343 25506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b414e2a197f14efbbd03d015ed9d3dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.094693 25650 tablet_service.cc:1467] Processing CreateTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 (DEFAULT_TABLE table=test-table [id=a264e4e016964916b0ec2b4709779c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:55.094784 25784 tablet_service.cc:1467] Processing CreateTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 (DEFAULT_TABLE table=test-table [id=a264e4e016964916b0ec2b4709779c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:55.095270 25650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b414e2a197f14efbbd03d015ed9d3dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.095554 25784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b414e2a197f14efbbd03d015ed9d3dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.100999 25875 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Bootstrap starting.
I20260927 22:14:55.102752 25875 tablet_bootstrap.cc:654] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.103209 25898 ts_tablet_manager.cc:1905] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.103890 25900 tablet_replica.cc:330] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: stopping tablet replica
I20260927 22:14:55.104169 25876 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Bootstrap starting.
I20260927 22:14:55.104735 25900 raft_consensus.cc:2229] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:14:55.105775 25876 tablet_bootstrap.cc:654] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.107347 25900 raft_consensus.cc:2258] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.111112 25875 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: No bootstrap required, opened a new log
I20260927 22:14:55.111271 25875 ts_tablet_manager.cc:1397] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:14:55.112027 25900 ts_tablet_manager.cc:1905] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.112079 25875 raft_consensus.cc:348] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.112299 25875 raft_consensus.cc:374] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.112401 25875 raft_consensus.cc:729] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad901bd6aa994c968aea6a808d1c6a89, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.115082 25875 consensus_queue.cc:260] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.116541 25876 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: No bootstrap required, opened a new log
I20260927 22:14:55.116703 25876 ts_tablet_manager.cc:1397] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260927 22:14:55.117035 25874 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Bootstrap starting.
I20260927 22:14:55.117719 25876 raft_consensus.cc:348] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.118062 25876 raft_consensus.cc:374] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.118160 25876 raft_consensus.cc:729] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ed0b1798c349abbf5ea0ab2527e3f2, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.118511 25876 consensus_queue.cc:260] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.118945 25874 tablet_bootstrap.cc:654] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.119927 25876 ts_tablet_manager.cc:1428] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 22:14:55.124009 25875 ts_tablet_manager.cc:1428] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:14:55.124557 25899 tablet_replica.cc:330] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: stopping tablet replica
I20260927 22:14:55.125033 25899 raft_consensus.cc:2229] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:14:55.125551 25899 raft_consensus.cc:2258] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.127562 25898 ts_tablet_manager.cc:1918] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:14:55.127884 25899 ts_tablet_manager.cc:1905] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.129401 25898 log.cc:1196] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/f90060711d6e46ada3c0109f3d6bf7ee
I20260927 22:14:55.132321 25874 tablet_bootstrap.cc:492] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: No bootstrap required, opened a new log
I20260927 22:14:55.132498 25874 ts_tablet_manager.cc:1397] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260927 22:14:55.133064 25898 ts_tablet_manager.cc:1939] T f90060711d6e46ada3c0109f3d6bf7ee P ad901bd6aa994c968aea6a808d1c6a89: Deleting consensus metadata
I20260927 22:14:55.134539 25874 raft_consensus.cc:348] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.134786 25874 raft_consensus.cc:374] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.134895 25874 raft_consensus.cc:729] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.134950 25899 ts_tablet_manager.cc:1918] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:14:55.135082 25899 log.cc:1196] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/f90060711d6e46ada3c0109f3d6bf7ee
I20260927 22:14:55.135130 25353 catalog_manager.cc:4841] TS ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937): tablet f90060711d6e46ada3c0109f3d6bf7ee (table test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]) successfully deleted
I20260927 22:14:55.135244 25874 consensus_queue.cc:260] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.135540 25899 ts_tablet_manager.cc:1939] T f90060711d6e46ada3c0109f3d6bf7ee P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting consensus metadata
I20260927 22:14:55.135829 25874 ts_tablet_manager.cc:1428] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:14:55.136415 25900 ts_tablet_manager.cc:1918] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:14:55.136628 25900 log.cc:1196] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/f90060711d6e46ada3c0109f3d6bf7ee
I20260927 22:14:55.137046 25355 catalog_manager.cc:4841] TS c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149): tablet f90060711d6e46ada3c0109f3d6bf7ee (table test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]) successfully deleted
I20260927 22:14:55.137322 25900 ts_tablet_manager.cc:1939] T f90060711d6e46ada3c0109f3d6bf7ee P 5e692b1469fb42768a17e4852c05914a: Deleting consensus metadata
I20260927 22:14:55.139045 25355 catalog_manager.cc:4841] TS 5e692b1469fb42768a17e4852c05914a (127.23.173.66:34353): tablet f90060711d6e46ada3c0109f3d6bf7ee (table test-table [id=33ae04bfea3b4d7e87d6cc4118286db5]) successfully deleted
I20260927 22:14:55.306748 25893 raft_consensus.cc:482] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:55.307080 25893 raft_consensus.cc:504] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.308049 25893 leader_election.cc:288] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937), c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149)
I20260927 22:14:55.308627 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b414e2a197f14efbbd03d015ed9d3dc8" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" is_pre_election: true
I20260927 22:14:55.308694 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b414e2a197f14efbbd03d015ed9d3dc8" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89" is_pre_election: true
I20260927 22:14:55.308887 25806 raft_consensus.cc:2454] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:55.308977 25531 raft_consensus.cc:2454] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:55.309392 25598 leader_election.cc:302] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, c9ed0b1798c349abbf5ea0ab2527e3f2; no voters:
I20260927 22:14:55.309880 25893 raft_consensus.cc:2781] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:55.310055 25893 raft_consensus.cc:482] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:55.310178 25893 raft_consensus.cc:3037] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.312636 25893 raft_consensus.cc:504] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.313217 25893 leader_election.cc:288] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Requested vote from peers ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937), c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149)
I20260927 22:14:55.313732 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b414e2a197f14efbbd03d015ed9d3dc8" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89"
I20260927 22:14:55.313825 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b414e2a197f14efbbd03d015ed9d3dc8" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
I20260927 22:14:55.313935 25531 raft_consensus.cc:3037] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.314069 25806 raft_consensus.cc:3037] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.316401 25531 raft_consensus.cc:2454] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:55.316855 25596 leader_election.cc:302] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, ad901bd6aa994c968aea6a808d1c6a89; no voters:
I20260927 22:14:55.317157 25893 raft_consensus.cc:2781] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:55.317329 25893 raft_consensus.cc:686] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 1 LEADER]: Becoming Leader. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Running, Role: LEADER
I20260927 22:14:55.317538 25806 raft_consensus.cc:2454] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:55.317670 25893 consensus_queue.cc:237] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.320538 25369 catalog_manager.cc:5495] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e692b1469fb42768a17e4852c05914a (127.23.173.66). New cstate: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:14:55 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547292, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:55.404026 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.388312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.23.173.126:42363)
0927 22:14:55.389363 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.389369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.389394 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:14:55.389450 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.389454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.389508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.389594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.391125 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.391863 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.393729 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.394237 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.394354 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.398486 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.398491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.398494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.398523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.401088 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.402032 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.402036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.402038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.402147 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.402544 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.402547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.402549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.402807 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.402911 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.403402 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.403736 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":917,"thread_start_us":794,"threads_started":1}
I20260927 22:14:55.404150 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.388277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42363 (local address 127.0.0.1:37328)
0927 22:14:55.389012 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:55.389036 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:55.389202 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:55.389679 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:55.389686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:55.390657 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:55.390976 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.390991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.392008 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.392014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.393553 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.393569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.394426 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.394433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.394637 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.395378 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:55.395409 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:55.398292 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:55.401218 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:55.401231 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:55.401239 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:55.401876 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:55.402254 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:55.402260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:55.402263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:55.402436 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:55.402929 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:55.402938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:55.403140 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 22:14:55.403327 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 22:14:55.403628 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":383,"threads_started":1}
I20260927 22:14:55.409284 25369 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37328:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:14:55.410454 25369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:55.417423 25506 tablet_service.cc:1467] Processing CreateTablet for tablet 5156b6f7bc684733a74967fd4780a4ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d5dabf1925c4be89560f1d36684794a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:55.417491 25650 tablet_service.cc:1467] Processing CreateTablet for tablet 5156b6f7bc684733a74967fd4780a4ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d5dabf1925c4be89560f1d36684794a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:55.418174 25650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5156b6f7bc684733a74967fd4780a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.418299 25506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5156b6f7bc684733a74967fd4780a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.418864 25785 tablet_service.cc:1467] Processing CreateTablet for tablet 5156b6f7bc684733a74967fd4780a4ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d5dabf1925c4be89560f1d36684794a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:55.419375 25785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5156b6f7bc684733a74967fd4780a4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:55.425215 25874 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: Bootstrap starting.
I20260927 22:14:55.425639 25875 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: Bootstrap starting.
I20260927 22:14:55.426285 25876 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: Bootstrap starting.
I20260927 22:14:55.427026 25874 tablet_bootstrap.cc:654] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.427417 25875 tablet_bootstrap.cc:654] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.428018 25876 tablet_bootstrap.cc:654] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:55.438246 25875 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: No bootstrap required, opened a new log
I20260927 22:14:55.438491 25875 ts_tablet_manager.cc:1397] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260927 22:14:55.438656 25876 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: No bootstrap required, opened a new log
I20260927 22:14:55.438850 25876 ts_tablet_manager.cc:1397] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260927 22:14:55.439172 25874 tablet_bootstrap.cc:492] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: No bootstrap required, opened a new log
I20260927 22:14:55.439352 25874 ts_tablet_manager.cc:1397] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260927 22:14:55.439721 25876 raft_consensus.cc:348] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.439985 25876 raft_consensus.cc:374] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.440076 25876 raft_consensus.cc:729] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ed0b1798c349abbf5ea0ab2527e3f2, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.440228 25874 raft_consensus.cc:348] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.440493 25874 raft_consensus.cc:374] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.440382 25876 consensus_queue.cc:260] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.440570 25874 raft_consensus.cc:729] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.440884 25882 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.440887 25874 consensus_queue.cc:260] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.441210 25882 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.441394 25893 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.441663 25893 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.441685 25876 ts_tablet_manager.cc:1428] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:14:55.442809 25874 ts_tablet_manager.cc:1428] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:55.443190 25875 raft_consensus.cc:348] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.443513 25875 raft_consensus.cc:374] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:55.443619 25875 raft_consensus.cc:729] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad901bd6aa994c968aea6a808d1c6a89, State: Initialized, Role: FOLLOWER
I20260927 22:14:55.443953 25875 consensus_queue.cc:260] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.444576 25875 ts_tablet_manager.cc:1428] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260927 22:14:55.445204 25881 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.445592 25881 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.485074 25893 raft_consensus.cc:482] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:55.485301 25893 raft_consensus.cc:504] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.486183 25893 leader_election.cc:288] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937), c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149)
I20260927 22:14:55.486720 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5156b6f7bc684733a74967fd4780a4ff" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" is_pre_election: true
I20260927 22:14:55.487017 25806 raft_consensus.cc:2454] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:55.487417 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5156b6f7bc684733a74967fd4780a4ff" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89" is_pre_election: true
I20260927 22:14:55.487511 25598 leader_election.cc:302] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, c9ed0b1798c349abbf5ea0ab2527e3f2; no voters:
I20260927 22:14:55.487676 25531 raft_consensus.cc:2454] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 0.
I20260927 22:14:55.487963 25893 raft_consensus.cc:2781] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:55.488085 25893 raft_consensus.cc:482] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:55.488222 25893 raft_consensus.cc:3037] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.490502 25893 raft_consensus.cc:504] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.491096 25893 leader_election.cc:288] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Requested vote from peers ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937), c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149)
I20260927 22:14:55.491449 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5156b6f7bc684733a74967fd4780a4ff" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad901bd6aa994c968aea6a808d1c6a89"
I20260927 22:14:55.491689 25531 raft_consensus.cc:3037] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.491613 25806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5156b6f7bc684733a74967fd4780a4ff" candidate_uuid: "5e692b1469fb42768a17e4852c05914a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2"
I20260927 22:14:55.491849 25806 raft_consensus.cc:3037] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:55.494752 25531 raft_consensus.cc:2454] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:55.494753 25806 raft_consensus.cc:2454] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e692b1469fb42768a17e4852c05914a in term 1.
I20260927 22:14:55.495203 25596 leader_election.cc:302] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e692b1469fb42768a17e4852c05914a, ad901bd6aa994c968aea6a808d1c6a89; no voters:
I20260927 22:14:55.495486 25893 raft_consensus.cc:2781] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:55.495666 25893 raft_consensus.cc:686] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [term 1 LEADER]: Becoming Leader. State: Replica: 5e692b1469fb42768a17e4852c05914a, State: Running, Role: LEADER
I20260927 22:14:55.495936 25893 consensus_queue.cc:237] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } }
I20260927 22:14:55.497606 25885 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: TxnStatusTablet state changed. Reason: New leader 5e692b1469fb42768a17e4852c05914a. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.497943 25885 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:55.498646 25916 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:14:55.498871 25916 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:14:55.499192 25369 catalog_manager.cc:5495] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e692b1469fb42768a17e4852c05914a (127.23.173.66). New cstate: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } health_report { overall_health: UNKNOWN } } }
I20260927 22:14:55.529810 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.524295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.23.173.66:34353)
0927 22:14:55.524479 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.524486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.524671 (+ 185us) server_negotiation.cc:407] Connection header received
0927 22:14:55.524888 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.524893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.525000 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.525151 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:55.525459 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.526520 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.527784 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.528509 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.528623 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.529024 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:55.529173 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:55.529253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.529352 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.529430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:14:55.529834 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.524288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34353 (local address 127.0.0.1:49300)
0927 22:14:55.524590 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:55.524606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:55.524734 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:55.525175 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:55.525178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:55.525187 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:14:55.525341 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.525348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.526637 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.526641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.527631 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.527641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.528652 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.528656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.528808 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.528861 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:14:55.529260 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:14:55.529275 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 22:14:55.529335 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 22:14:55.529535 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 22:14:55.533793 25806 raft_consensus.cc:1261] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.533891 25531 raft_consensus.cc:1261] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.534610 25893 consensus_queue.cc:1035] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.534900 25885 consensus_queue.cc:1035] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.537670 25882 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: TxnStatusTablet state changed. Reason: New leader 5e692b1469fb42768a17e4852c05914a. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.537932 25882 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.539539 25885 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.539808 25885 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:55.539989 25888 mvcc.cc:205] Tried to move back new op lower bound from 7334081722500100096 to 7334081722353733632. Current Snapshot: MvccSnapshot[applied={T|T < 7334081722500100096 or (T in {7334081722500100096})}]
I20260927 22:14:55.541644 25881 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: TxnStatusTablet state changed. Reason: New leader 5e692b1469fb42768a17e4852c05914a. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.541918 25881 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.542338 25880 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.542591 25880 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P 5e692b1469fb42768a17e4852c05914a: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:55.543355 25889 mvcc.cc:205] Tried to move back new op lower bound from 7334081722500100096 to 7334081722353733632. Current Snapshot: MvccSnapshot[applied={T|T < 7334081722500100096}]
I20260927 22:14:55.543644 25882 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.543871 25882 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P c9ed0b1798c349abbf5ea0ab2527e3f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.544595 25890 mvcc.cc:205] Tried to move back new op lower bound from 7334081722500100096 to 7334081722353733632. Current Snapshot: MvccSnapshot[applied={T|T < 7334081722500100096}]
I20260927 22:14:55.545243 25881 tablet_replica.cc:439] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e692b1469fb42768a17e4852c05914a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 } } peers { permanent_uuid: "5e692b1469fb42768a17e4852c05914a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34353 } } peers { permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 } } }
I20260927 22:14:55.545504 25881 tablet_replica.cc:442] T 5156b6f7bc684733a74967fd4780a4ff P ad901bd6aa994c968aea6a808d1c6a89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:55.569126 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.562574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38181 (local address 127.23.173.66:34353)
0927 22:14:55.562809 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.562815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.563425 (+ 610us) server_negotiation.cc:407] Connection header received
0927 22:14:55.563591 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.563598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.563696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.563806 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:55.564373 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.565505 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.567152 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.567849 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.567984 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.568414 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:55.568537 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:55.568623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.568746 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.568836 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:14:55.569128 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.562610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34353 (local address 127.23.173.66:38181)
0927 22:14:55.563325 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:55.563341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:55.563473 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:55.563810 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:55.563813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:55.563821 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:14:55.564230 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.564239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.565618 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.565621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.567015 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:55.567026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.568026 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.568030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:55.568167 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.568220 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:14:55.568638 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:14:55.568646 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:14:55.568700 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 22:14:55.568800 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":27,"thread_start_us":403,"threads_started":1}
I20260927 22:14:55.580566 25531 raft_consensus.cc:1261] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.580565 25806 raft_consensus.cc:1261] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Refusing update from remote peer 5e692b1469fb42768a17e4852c05914a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:55.581190 25885 consensus_queue.cc:1035] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad901bd6aa994c968aea6a808d1c6a89" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.581533 25893 consensus_queue.cc:1035] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ed0b1798c349abbf5ea0ab2527e3f2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:55.645246 25367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:14:55.645478 25367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:14:55.648821 25367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a7bba45b35d4d7d8e051e05312bfadf: Sending DeleteTablet for 3 replicas of tablet b414e2a197f14efbbd03d015ed9d3dc8
I20260927 22:14:55.649995 25506 tablet_service.cc:1514] Processing DeleteTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:39484
I20260927 22:14:55.650398 25785 tablet_service.cc:1514] Processing DeleteTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:46388
I20260927 22:14:55.653429 25651 tablet_service.cc:1514] Processing DeleteTablet for tablet b414e2a197f14efbbd03d015ed9d3dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:14:55 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:49290
I20260927 22:14:55.684821 25929 tablet_replica.cc:330] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: stopping tablet replica
I20260927 22:14:55.685312 25929 raft_consensus.cc:2229] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:14:55.686157 25929 raft_consensus.cc:2258] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.688016 25929 ts_tablet_manager.cc:1905] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.691470 25927 tablet_replica.cc:330] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: stopping tablet replica
I20260927 22:14:55.691802 25927 raft_consensus.cc:2229] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:14:55.692260 25926 tablet_replica.cc:330] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: stopping tablet replica
I20260927 22:14:55.692540 25927 raft_consensus.cc:2258] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.694209 25926 raft_consensus.cc:2229] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:14:55.694636 25926 raft_consensus.cc:2258] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:14:55.694967 25927 ts_tablet_manager.cc:1905] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.696045 25926 ts_tablet_manager.cc:1905] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:14:55.699859 25929 ts_tablet_manager.cc:1918] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:14:55.700071 25929 log.cc:1196] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/b414e2a197f14efbbd03d015ed9d3dc8
I20260927 22:14:55.700778 25929 ts_tablet_manager.cc:1939] T b414e2a197f14efbbd03d015ed9d3dc8 P 5e692b1469fb42768a17e4852c05914a: Deleting consensus metadata
I20260927 22:14:55.702455 25355 catalog_manager.cc:4841] TS 5e692b1469fb42768a17e4852c05914a (127.23.173.66:34353): tablet b414e2a197f14efbbd03d015ed9d3dc8 (table test-table [id=a264e4e016964916b0ec2b4709779c17]) successfully deleted
I20260927 22:14:55.714607 25927 ts_tablet_manager.cc:1918] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:14:55.715119 25926 ts_tablet_manager.cc:1918] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:14:55.715379 25927 log.cc:1196] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/b414e2a197f14efbbd03d015ed9d3dc8
I20260927 22:14:55.715938 25926 log.cc:1196] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/b414e2a197f14efbbd03d015ed9d3dc8
I20260927 22:14:55.716495 25926 ts_tablet_manager.cc:1939] T b414e2a197f14efbbd03d015ed9d3dc8 P ad901bd6aa994c968aea6a808d1c6a89: Deleting consensus metadata
I20260927 22:14:55.716749 25927 ts_tablet_manager.cc:1939] T b414e2a197f14efbbd03d015ed9d3dc8 P c9ed0b1798c349abbf5ea0ab2527e3f2: Deleting consensus metadata
I20260927 22:14:55.718189 25355 catalog_manager.cc:4841] TS c9ed0b1798c349abbf5ea0ab2527e3f2 (127.23.173.67:39149): tablet b414e2a197f14efbbd03d015ed9d3dc8 (table test-table [id=a264e4e016964916b0ec2b4709779c17]) successfully deleted
I20260927 22:14:55.719077 25353 catalog_manager.cc:4841] TS ad901bd6aa994c968aea6a808d1c6a89 (127.23.173.65:45937): tablet b414e2a197f14efbbd03d015ed9d3dc8 (table test-table [id=a264e4e016964916b0ec2b4709779c17]) successfully deleted
Sep 27 22:14:55 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:14:55.737358 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.714248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.23.173.65:45937)
0927 22:14:55.720170 (+ 5922us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.720179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.720201 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:14:55.720274 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.720280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.720363 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.720482 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.722843 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.723670 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.725734 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.726286 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.726391 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.731496 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.731513 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.731531 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.731572 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.734546 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.735262 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.735270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.735272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.735424 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.735803 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.735807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.735809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.736182 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.736496 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.736783 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.737015 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5768,"thread_start_us":5622,"threads_started":1}
W20260927 22:14:55.738149 25571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39490
I20260927 22:14:55.755795 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.741352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.23.173.126:42363)
0927 22:14:55.741630 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.741637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.741932 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:14:55.742227 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.742231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.742293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.742366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.744151 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.744849 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.746810 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.747208 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.747299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.750519 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.750524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.750541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.750571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.753133 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.754043 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.754047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.754049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.754156 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.754525 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.754528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.754530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.754799 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.754983 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.755363 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.755532 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
W20260927 22:14:55.756356 25367 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330
I20260927 22:14:55.758224 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25440
I20260927 22:14:55.771682 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.759117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.23.173.65:45937)
0927 22:14:55.759325 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.759333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.760151 (+ 818us) server_negotiation.cc:407] Connection header received
0927 22:14:55.760343 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.760348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.760427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.760536 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.761724 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.762593 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.763754 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.764191 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.764326 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.766501 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.766507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.766520 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.766565 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.769070 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.769667 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.769674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.769678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.769806 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.770181 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.770188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.770191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.770451 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.770716 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.771009 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.771239 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:14:55.772218 25571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39502
W20260927 22:14:55.772737 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25440: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:55.772905 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25440
I20260927 22:14:55.798517 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25580
Sep 27 22:14:55 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:14:55.817049 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.799419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.23.173.66:34353)
0927 22:14:55.799593 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.799597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.799615 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:14:55.799842 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.799846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.799908 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.800011 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.801691 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.802526 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.804485 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.804938 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.805062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.810344 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.810358 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.810373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.810407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.813496 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.814218 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.814226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.814228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.814376 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.814833 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.814838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.814842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.815233 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.815574 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.816460 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.816762 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:14:55.817878 25711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49308
W20260927 22:14:55.818567 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:55.818758 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25580
I20260927 22:14:55.841822 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25720
Sep 27 22:14:55 dist-test-slave-3wc3 krb5kdc[25324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:14:55.856720 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:55.842644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.23.173.67:39149)
0927 22:14:55.843131 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:55.843135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:55.843152 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:14:55.843210 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:55.843214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:55.843278 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:55.843372 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:55.844517 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.845314 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.846586 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:55.847002 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:55.847093 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:55.851148 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:55.851163 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:55.851179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:55.851213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:55.854296 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.854924 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.854931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.854933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.855096 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:55.855390 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:55.855393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:55.855395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:55.855715 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:55.855930 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:55.856167 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:14:55.856342 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
W20260927 22:14:55.857430 25850 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46392
W20260927 22:14:55.858141 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:14:55.858295 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25720
I20260927 22:14:55.874861 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25337
I20260927 22:14:55.875985 25409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:14:55.990727 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25337
2026-09-27T22:14:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4716 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:14:56 dist-test-slave-3wc3 krb5kdc[25939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:14:56 dist-test-slave-3wc3 krb5kdc[25939](info): set up 2 sockets
Sep 27 22:14:56 dist-test-slave-3wc3 krb5kdc[25939](info): commencing operation
krb5kdc: starting...
W20260927 22:14:57.090215 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:14:57 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:14:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:14:57.278187 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35409
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:57.591521 25952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:57.591859 25952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:57.592003 25952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:57.599480 25952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:14:57.599622 25952 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:14:57.599718 25952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:57.599838 25952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:57.599924 25952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:14:57.600013 25952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:14:57.609856 25952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:35409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:57.611196 25952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:57.612540 25952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:57.624347 25957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:57.624286 25958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:57.624593 25960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:57.625270 25952 server_base.cc:1034] running on GCE node
I20260927 22:14:57.625941 25952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:57.627259 25952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:57.628500 25952 hybrid_clock.cc:648] HybridClock initialized: now 1790547297628430 us; error 45 us; skew 500 ppm
I20260927 22:14:57.628813 25952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:57 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547297, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:57.631996 25952 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:14:57.633495 25952 webserver.cc:460] Webserver started at http://127.23.173.126:41965/ using document root <none> and password file <none>
I20260927 22:14:57.634315 25952 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:57.634425 25952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:57.634756 25952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:57.637676 25952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "9e18c89f2b614ceb931334342c0a844f"
format_stamp: "Formatted at 2026-09-27 22:14:57 on dist-test-slave-3wc3"
server_key: "5ac6f7f0a9c2198990e152fe609dc4bd"
server_key_iv: "eef34689394cff003b97a5a2337f6da4"
server_key_version: "encryptionkey@0"
I20260927 22:14:57.638477 25952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "9e18c89f2b614ceb931334342c0a844f"
format_stamp: "Formatted at 2026-09-27 22:14:57 on dist-test-slave-3wc3"
server_key: "5ac6f7f0a9c2198990e152fe609dc4bd"
server_key_iv: "eef34689394cff003b97a5a2337f6da4"
server_key_version: "encryptionkey@0"
I20260927 22:14:57.643560 25952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:57.646983 25967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:57.648219 25952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:57.648380 25952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9e18c89f2b614ceb931334342c0a844f"
format_stamp: "Formatted at 2026-09-27 22:14:57 on dist-test-slave-3wc3"
server_key: "5ac6f7f0a9c2198990e152fe609dc4bd"
server_key_iv: "eef34689394cff003b97a5a2337f6da4"
server_key_version: "encryptionkey@0"
I20260927 22:14:57.648594 25952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:57.659611 25952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:57.663304 25952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:57.663565 25952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:57.685083 25952 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:35409
I20260927 22:14:57.685102 26028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:35409 every 8 connection(s)
I20260927 22:14:57.686455 25952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:14:57.690104 26029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:57.694715 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 25952
I20260927 22:14:57.694916 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:14:57.695729 24245 external_mini_cluster.cc:1428] Setting key 70ecddda83e833a3bacb78d44ab7ee97
I20260927 22:14:57.702246 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Bootstrap starting.
I20260927 22:14:57.705765 26029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:57.706959 26029 log.cc:824] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:57.710287 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: No bootstrap required, opened a new log
Sep 27 22:14:57 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:57.719894 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:57.697806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.23.173.126:35409)
0927 22:14:57.698624 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:57.698638 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:57.698678 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:14:57.700485 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:57.700507 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:57.700882 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:57.701196 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:57.702988 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:57.704310 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:57.706121 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:57.706811 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:57.706939 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:57.711557 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:57.711568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:57.711586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:57.711632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:57.715535 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:57.716174 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:57.716182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:57.716190 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:57.716414 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:57.716741 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:57.716746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:57.716750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:57.717433 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:57.717737 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:57.718075 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:14:57.718506 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":394,"threads_started":1}
I20260927 22:14:57.720942 26029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } }
I20260927 22:14:57.721302 26029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:57.721439 26029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e18c89f2b614ceb931334342c0a844f, State: Initialized, Role: FOLLOWER
I20260927 22:14:57.722213 26029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } }
I20260927 22:14:57.722457 26029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:14:57.722589 26029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:14:57.722731 26029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:57.724844 26029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } }
I20260927 22:14:57.725345 26029 leader_election.cc:302] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e18c89f2b614ceb931334342c0a844f; no voters:
I20260927 22:14:57.725878 26029 leader_election.cc:288] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:14:57.726042 26034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:57.726718 26034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [term 1 LEADER]: Becoming Leader. State: Replica: 9e18c89f2b614ceb931334342c0a844f, State: Running, Role: LEADER
I20260927 22:14:57.727258 26034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } }
I20260927 22:14:57.728173 26029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:14:57.730540 26035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e18c89f2b614ceb931334342c0a844f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } } }
I20260927 22:14:57.730615 26036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e18c89f2b614ceb931334342c0a844f. Latest consensus state: current_term: 1 leader_uuid: "9e18c89f2b614ceb931334342c0a844f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e18c89f2b614ceb931334342c0a844f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35409 } } }
I20260927 22:14:57.731091 26035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:57.731118 26036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f [sys.catalog]: This master's current role is: LEADER
I20260927 22:14:57.737118 26043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:14:57.742533 26043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:14:57.751271 26043 catalog_manager.cc:1347] Generated new cluster ID: 9b9f7e3c582846d0a2241046e83eb376
I20260927 22:14:57.751377 26043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:14:57.777894 26043 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:14:57.779879 26043 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:14:57.789103 26043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Generated new TSK 0
I20260927 22:14:57.790123 26043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:14:57.850221 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35409
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:58.177377 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:58.177651 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:58.177819 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:58.185096 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:58.185192 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:58.185274 26055 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:14:58.185328 26055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:58.185388 26055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:14:58.197844 26055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:58.199347 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:58.200713 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:58.216683 26061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:58.217396 26063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:58.216707 26060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:58.218752 26055 server_base.cc:1034] running on GCE node
I20260927 22:14:58.219559 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:58.220916 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:58.222163 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790547298222100 us; error 43 us; skew 500 ppm
I20260927 22:14:58.222584 26055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:58 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:58.226846 26055 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:14:58.228343 26055 webserver.cc:460] Webserver started at http://127.23.173.65:41851/ using document root <none> and password file <none>
I20260927 22:14:58.229077 26055 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:58.229216 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:58.229532 26055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:58.232337 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "bd154cd665be48f5b701775fed332f14"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "56871d5faa35fc728e01663b46c99d7e"
server_key_iv: "dd9e360dd5c12bd9e91ec8b4455f5c20"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.233232 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "bd154cd665be48f5b701775fed332f14"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "56871d5faa35fc728e01663b46c99d7e"
server_key_iv: "dd9e360dd5c12bd9e91ec8b4455f5c20"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.242496 26055 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.005s
I20260927 22:14:58.246191 26070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:58.247417 26055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:14:58.247550 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "bd154cd665be48f5b701775fed332f14"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "56871d5faa35fc728e01663b46c99d7e"
server_key_iv: "dd9e360dd5c12bd9e91ec8b4455f5c20"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.247706 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:58.264365 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:58.268484 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:58.268698 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:58.271234 26055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:58.271368 26055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:58.271507 26055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:58.271598 26055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:58 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:58.295591 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.271658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:35357 (local address 127.23.173.126:35409)
0927 22:14:58.272458 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:58.272465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:58.273378 (+ 913us) server_negotiation.cc:407] Connection header received
0927 22:14:58.275671 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:58.275676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:58.275771 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:58.275886 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:58.278487 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.279304 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.281884 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.282579 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.282717 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.287779 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:58.287787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:58.287793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:58.287843 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:58.291632 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.292837 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.292845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.292852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.293029 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.293626 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.293632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.293635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.293913 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:58.294107 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:58.294985 (+ 878us) server_negotiation.cc:299] Negotiation successful
0927 22:14:58.295213 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":567,"threads_started":1}
I20260927 22:14:58.296840 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.272298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.65:35357)
0927 22:14:58.273208 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:58.273253 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:58.275388 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:58.276228 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:58.276239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:58.277237 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:58.278258 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.278295 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.279495 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.279506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.281684 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.281699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.282838 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.282847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.283170 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.284028 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:58.284072 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:58.287520 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:58.291845 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.291858 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.291884 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.292642 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.293211 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.293217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.293227 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.293488 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.294081 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:58.294088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:58.294551 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 22:14:58.294903 (+ 352us) client_negotiation.cc:240] Negotiation successful
0927 22:14:58.295448 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":425,"threads_started":1}
I20260927 22:14:58.319068 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.303205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.65:55005)
0927 22:14:58.303831 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:58.303857 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:58.303976 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:58.304405 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:58.304409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:58.304869 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:58.305152 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.305160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.306881 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.306884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.307932 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.307943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.309149 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.309154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.309284 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.310256 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:58.310287 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:58.311831 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:58.315478 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.315487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.315491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.316219 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.316656 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.316664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.316668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.316928 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.317630 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:58.317636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:58.317833 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 22:14:58.318052 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:14:58.318449 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":323,"threads_started":1}
I20260927 22:14:58.319491 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.303093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:55005 (local address 127.23.173.126:35409)
0927 22:14:58.303337 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:58.303346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:58.303952 (+ 606us) server_negotiation.cc:407] Connection header received
0927 22:14:58.304113 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:58.304121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:58.304227 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:58.304357 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:58.305294 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.306732 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.308071 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.308928 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.309078 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.312068 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:58.312078 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:58.312083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:58.312131 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:58.315248 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.316364 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.316372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.316377 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.316537 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.317079 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.317084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.317087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.317397 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:58.317558 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:58.319114 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0927 22:14:58.319277 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:14:58.321187 26055 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:42895
I20260927 22:14:58.321206 26190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:42895 every 8 connection(s)
I20260927 22:14:58.322795 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:14:58.324971 26191 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35409
I20260927 22:14:58.325166 26191 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:58.325745 26191 heartbeater.cc:511] Master 127.23.173.126:35409 requested a full tablet report, sending...
I20260927 22:14:58.327601 25984 ts_manager.cc:194] Registered new tserver with Master: bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895)
I20260927 22:14:58.329200 25984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:35357
I20260927 22:14:58.332624 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26055
I20260927 22:14:58.332854 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:14:58.333505 24245 external_mini_cluster.cc:1428] Setting key 7cad3775801fd658a42b4c116ce3b754
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:14:58.371420 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35409
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:58.682467 26195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:58.682714 26195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:58.682809 26195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:58.692889 26195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:58.693018 26195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:58.693149 26195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:14:58.693216 26195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:58.693312 26195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:14:58.708351 26195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:58.709625 26195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:58.710932 26195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:58.723819 26200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:58.723778 26203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:58.723865 26201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:58.724727 26195 server_base.cc:1034] running on GCE node
I20260927 22:14:58.725328 26195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:58.726325 26195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:58.727568 26195 hybrid_clock.cc:648] HybridClock initialized: now 1790547298727525 us; error 60 us; skew 500 ppm
I20260927 22:14:58.727952 26195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:58 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:58.731215 26195 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:14:58.732620 26195 webserver.cc:460] Webserver started at http://127.23.173.66:35535/ using document root <none> and password file <none>
I20260927 22:14:58.733259 26195 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:58.733356 26195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:58.733626 26195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:58.736716 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "a6aecc557fca4b7e9c283ed937977969"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "2d4d6849f2571c26665222719dca9e8c"
server_key_iv: "06efcfbdabbe4c9af39b668a066d910f"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.737546 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "a6aecc557fca4b7e9c283ed937977969"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "2d4d6849f2571c26665222719dca9e8c"
server_key_iv: "06efcfbdabbe4c9af39b668a066d910f"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.742841 26195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:14:58.746429 26210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:58.747764 26195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:14:58.747951 26195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "a6aecc557fca4b7e9c283ed937977969"
format_stamp: "Formatted at 2026-09-27 22:14:58 on dist-test-slave-3wc3"
server_key: "2d4d6849f2571c26665222719dca9e8c"
server_key_iv: "06efcfbdabbe4c9af39b668a066d910f"
server_key_version: "encryptionkey@0"
I20260927 22:14:58.748153 26195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:58.789057 26195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:58.792970 26195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:58.793207 26195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:58.795897 26195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:58.796031 26195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:58.796164 26195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:58.796242 26195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:58 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:58.820361 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.796234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36645 (local address 127.23.173.126:35409)
0927 22:14:58.796416 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:58.796438 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:58.798013 (+ 1575us) server_negotiation.cc:407] Connection header received
0927 22:14:58.800537 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:58.800545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:58.800702 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:58.800823 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:58.803337 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.805023 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.807682 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.808371 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.808533 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.813784 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:58.813789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:58.813792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:58.813832 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:58.816551 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.817628 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.817632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.817634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.817755 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.818379 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.818386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.818393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.818657 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:58.818828 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:58.819771 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 22:14:58.820014 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:14:58.822176 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.796941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.66:36645)
0927 22:14:58.797800 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:58.797854 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:58.800118 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:58.801175 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:58.801187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:58.802268 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:58.803100 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.803129 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.805204 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.805210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.807447 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.807460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.808683 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.808691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.808885 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.809777 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:58.809818 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:58.813488 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:58.816689 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.816701 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.816724 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.817495 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.817892 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.817897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.817901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.818238 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.818795 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:58.818804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:58.819282 (+ 478us) client_negotiation.cc:769] Sending connection context
0927 22:14:58.819666 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 22:14:58.820198 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":339,"threads_started":1}
I20260927 22:14:58.843387 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.828456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:51817 (local address 127.23.173.126:35409)
0927 22:14:58.828625 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:58.828629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:58.829275 (+ 646us) server_negotiation.cc:407] Connection header received
0927 22:14:58.829496 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:58.829501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:58.829571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:58.829668 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:58.830906 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.831619 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.832890 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.833326 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.833465 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.835838 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:58.835844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:58.835850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:58.835891 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:58.840143 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.841009 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.841017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.841020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.841167 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:58.841675 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:58.841683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:58.841688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:58.842228 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:58.842390 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:58.842800 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 22:14:58.843033 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:14:58.843488 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:58.828522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.66:51817)
0927 22:14:58.829190 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:58.829204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:58.829357 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:58.829690 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:58.829695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:58.830245 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:58.830754 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.830772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.831738 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.831745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.832752 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:58.832763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:58.833513 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:58.833519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:58.833696 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:58.834404 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:58.834433 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:58.835538 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:58.840286 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.840289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.840292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.840879 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.841338 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:58.841346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:58.841352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:58.841550 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:58.842353 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:58.842358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:58.842550 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 22:14:58.842690 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:14:58.843022 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":361,"threads_started":1}
I20260927 22:14:58.854733 26195 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:38291
I20260927 22:14:58.854774 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:38291 every 8 connection(s)
I20260927 22:14:58.855947 26195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:14:58.857801 26330 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35409
I20260927 22:14:58.858035 26330 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:58.858635 26330 heartbeater.cc:511] Master 127.23.173.126:35409 requested a full tablet report, sending...
I20260927 22:14:58.859144 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26195
I20260927 22:14:58.859324 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:14:58.859827 24245 external_mini_cluster.cc:1428] Setting key 07674263d87d360c4c78085bb7e0b4a6
I20260927 22:14:58.860176 25984 ts_manager.cc:194] Registered new tserver with Master: a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291)
I20260927 22:14:58.861320 25984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:36645
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:14:58.899554 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35409
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:14:59.215404 26334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:14:59.215652 26334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:14:59.215751 26334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:14:59.222568 26334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:14:59.222659 26334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:14:59.222739 26334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:14:59.222790 26334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:14:59.222855 26334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:14:59.232400 26334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:14:59.233598 26334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:14:59.234995 26334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:14:59.247807 26339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:59.247831 26342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:14:59.247984 26340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:14:59.248977 26334 server_base.cc:1034] running on GCE node
I20260927 22:14:59.249575 26334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:14:59.250622 26334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:14:59.251875 26334 hybrid_clock.cc:648] HybridClock initialized: now 1790547299251767 us; error 56 us; skew 500 ppm
I20260927 22:14:59.252197 26334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:14:59 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547299, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:14:59.262764 26334 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:14:59.264173 26334 webserver.cc:460] Webserver started at http://127.23.173.67:33487/ using document root <none> and password file <none>
I20260927 22:14:59.264796 26334 fs_manager.cc:359] Metadata directory not provided
I20260927 22:14:59.264904 26334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:14:59.265156 26334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:14:59.268018 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "71f9c41a406e41e0af6f7120dae2936f"
format_stamp: "Formatted at 2026-09-27 22:14:59 on dist-test-slave-3wc3"
server_key: "d6d453e405930da133aa6b4809c015b1"
server_key_iv: "794d2a9933e8caba4cc6c5b5c49d61ea"
server_key_version: "encryptionkey@0"
I20260927 22:14:59.268692 26334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "71f9c41a406e41e0af6f7120dae2936f"
format_stamp: "Formatted at 2026-09-27 22:14:59 on dist-test-slave-3wc3"
server_key: "d6d453e405930da133aa6b4809c015b1"
server_key_iv: "794d2a9933e8caba4cc6c5b5c49d61ea"
server_key_version: "encryptionkey@0"
I20260927 22:14:59.273592 26334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:14:59.276866 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:59.278339 26334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:59.278468 26334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "71f9c41a406e41e0af6f7120dae2936f"
format_stamp: "Formatted at 2026-09-27 22:14:59 on dist-test-slave-3wc3"
server_key: "d6d453e405930da133aa6b4809c015b1"
server_key_iv: "794d2a9933e8caba4cc6c5b5c49d61ea"
server_key_version: "encryptionkey@0"
I20260927 22:14:59.278620 26334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:14:59.293277 26334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:14:59.297014 26334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:14:59.297233 26334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:14:59.299810 26334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:14:59.299922 26334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:14:59.300068 26334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:14:59.300151 26334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:14:59 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547299, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:59.321259 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.300433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:38495 (local address 127.23.173.126:35409)
0927 22:14:59.300640 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.300647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.302044 (+ 1397us) server_negotiation.cc:407] Connection header received
0927 22:14:59.304041 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.304047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.304149 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.304260 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:59.306361 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.307115 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.308664 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.309071 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.309200 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.313761 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:59.313771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:59.313778 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:59.313841 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:59.317813 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.318862 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.318867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.318871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.319011 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.319442 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.319447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.319450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.319709 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:59.319884 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.320628 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.320908 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:14:59.322399 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.300985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.67:38495)
0927 22:14:59.301862 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.301916 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.303765 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.304527 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.304541 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.305325 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:59.306181 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.306200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.307270 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.307275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.308525 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.308537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.309309 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.309316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.309511 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.310305 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:59.310339 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:59.313536 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:59.318014 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.318024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.318046 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.318735 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.319123 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.319128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.319132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.319324 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.319842 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:59.319850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:59.320230 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.320486 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.320971 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":435,"threads_started":1}
I20260927 22:14:59.331712 26191 heartbeater.cc:503] Master 127.23.173.126:35409 was elected leader, sending a full tablet report...
I20260927 22:14:59.353542 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.336708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.23.173.67:36995)
0927 22:14:59.337458 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.337479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.337620 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.338031 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.338036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.338840 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:59.339414 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.339439 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.340935 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.340940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.342175 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.342197 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.343583 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.343588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.343839 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.344652 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:59.344690 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:59.346091 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:59.350440 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.350445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.350449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.350981 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.351463 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.351469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.351473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.351636 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.352365 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:59.352370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:59.352559 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.352722 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.353119 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":365,"threads_started":1}
I20260927 22:14:59.354350 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.336652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:36995 (local address 127.23.173.126:35409)
0927 22:14:59.336831 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.336835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.337549 (+ 714us) server_negotiation.cc:407] Connection header received
0927 22:14:59.337766 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.337771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.337853 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.337968 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:59.339587 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.340765 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.342392 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.343135 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.343303 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.346354 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:59.346359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:59.346362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:59.346400 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:59.350239 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.351138 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.351144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.351148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.351353 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.351846 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.351852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.351856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.352240 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:59.352413 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.353893 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.354123 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:14:59.365209 26334 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:37359
I20260927 22:14:59.365293 26468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:37359 every 8 connection(s)
I20260927 22:14:59.366508 26334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:14:59.368716 26469 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35409
I20260927 22:14:59.368950 26469 heartbeater.cc:464] Registering TS with master...
I20260927 22:14:59.369563 26469 heartbeater.cc:511] Master 127.23.173.126:35409 requested a full tablet report, sending...
I20260927 22:14:59.370983 25984 ts_manager.cc:194] Registered new tserver with Master: 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:14:59.371773 25984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:38495
I20260927 22:14:59.376530 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26334
I20260927 22:14:59.376667 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:14:59.377053 24245 external_mini_cluster.cc:1428] Setting key fcfe79ce2fb9278b1980416223ea3f9b
I20260927 22:14:59.379943 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:14:59 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:14:59 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:59.426030 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.412611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.23.173.126:35409)
0927 22:14:59.412811 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.412815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.413199 (+ 384us) server_negotiation.cc:407] Connection header received
0927 22:14:59.413349 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.413353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.413417 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.413510 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:59.415021 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.415723 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.416935 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.417351 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.417439 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.421136 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:59.421141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:59.421144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:59.421173 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:59.423591 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.424284 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.424288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.424290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.424412 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.424793 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.424796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.424798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.425051 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:59.425232 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.425537 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.425691 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:14:59.430611 25984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:14:59.438477 25984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:59.461589 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.451871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38291 (local address 127.0.0.1:35810)
0927 22:14:59.452606 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.452632 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.452755 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.453577 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.453585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.453636 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.454279 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.454298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.456426 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.456432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.458617 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.458629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.460425 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.460433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.460663 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.460731 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.460870 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.461080 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":42,"spinlock_wait_cycles":7808,"thread_start_us":273,"threads_started":1}
I20260927 22:14:59.462051 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.450761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.23.173.65:42895)
0927 22:14:59.451372 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.451382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.451694 (+ 312us) server_negotiation.cc:407] Connection header received
0927 22:14:59.452853 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.452866 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.453131 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.453317 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.455059 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.456631 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.459392 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.460510 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.460656 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.460889 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.461458 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.461659 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":323,"threads_started":1}
I20260927 22:14:59.462173 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.452081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.23.173.66:38291)
0927 22:14:59.452702 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.452710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.452739 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:14:59.452957 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.452965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.453153 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.453340 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.454564 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.456297 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.458768 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.460181 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.460316 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.460523 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.461574 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.461863 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":279,"threads_started":1}
I20260927 22:14:59.463213 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.450557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.0.0.1:33202)
0927 22:14:59.451526 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.451587 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.451765 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.453580 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.453587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.453628 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.454864 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.454877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.456825 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.456838 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.459226 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.459243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.461056 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.461063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.461235 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.461271 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.462752 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.462853 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":819,"spinlock_wait_cycles":896,"thread_start_us":462,"threads_started":1}
I20260927 22:14:59.463593 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.453094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.23.173.67:37359)
0927 22:14:59.455606 (+ 2512us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.455612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.455636 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:14:59.455704 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.455709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.455867 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.456050 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.456796 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.458719 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.461054 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.462472 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.462568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.462707 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.463093 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.463262 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2381,"thread_start_us":2187,"threads_started":1}
I20260927 22:14:59.465276 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.452500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37359 (local address 127.0.0.1:54156)
0927 22:14:59.454254 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.454274 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.454397 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.456051 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.456055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.456084 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.456531 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.456548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.458870 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.458875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.460868 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.460880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.462674 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.462682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.462885 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.462920 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.463032 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.463166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1562,"mutex_wait_us":22,"thread_start_us":1376,"threads_started":1}
I20260927 22:14:59.466475 26260 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 (DEFAULT_TABLE table=test-table [id=d724cfdac2da4747bf3824574f02892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:59.466419 26121 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 (DEFAULT_TABLE table=test-table [id=d724cfdac2da4747bf3824574f02892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:59.467716 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01a3ba39dc4707938a56860b8b6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.467777 26121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01a3ba39dc4707938a56860b8b6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.469030 26401 tablet_service.cc:1467] Processing CreateTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 (DEFAULT_TABLE table=test-table [id=d724cfdac2da4747bf3824574f02892e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:14:59.470424 26401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e01a3ba39dc4707938a56860b8b6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.479684 26488 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Bootstrap starting.
I20260927 22:14:59.481760 26489 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Bootstrap starting.
I20260927 22:14:59.484464 26490 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Bootstrap starting.
I20260927 22:14:59.484972 26488 tablet_bootstrap.cc:654] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.485350 26489 tablet_bootstrap.cc:654] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.486460 26488 log.cc:824] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:59.486546 26489 log.cc:824] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:59.487330 26490 tablet_bootstrap.cc:654] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.488591 26490 log.cc:824] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:14:59.489604 26489 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: No bootstrap required, opened a new log
I20260927 22:14:59.489904 26489 ts_tablet_manager.cc:1397] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:14:59.490547 26488 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: No bootstrap required, opened a new log
I20260927 22:14:59.490833 26488 ts_tablet_manager.cc:1397] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 22:14:59.491062 26490 tablet_bootstrap.cc:492] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: No bootstrap required, opened a new log
I20260927 22:14:59.491293 26490 ts_tablet_manager.cc:1397] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 22:14:59.498266 26490 raft_consensus.cc:348] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.498622 26490 raft_consensus.cc:374] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.498713 26490 raft_consensus.cc:729] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.499360 26490 consensus_queue.cc:260] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.500594 26469 heartbeater.cc:503] Master 127.23.173.126:35409 was elected leader, sending a full tablet report...
I20260927 22:14:59.500917 26490 ts_tablet_manager.cc:1428] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260927 22:14:59.503013 26488 raft_consensus.cc:348] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.503013 26489 raft_consensus.cc:348] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.503424 26488 raft_consensus.cc:374] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.503424 26489 raft_consensus.cc:374] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.503533 26489 raft_consensus.cc:729] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd154cd665be48f5b701775fed332f14, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.503525 26488 raft_consensus.cc:729] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.504125 26488 consensus_queue.cc:260] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.504122 26489 consensus_queue.cc:260] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.505175 26330 heartbeater.cc:503] Master 127.23.173.126:35409 was elected leader, sending a full tablet report...
I20260927 22:14:59.505827 26488 ts_tablet_manager.cc:1428] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 22:14:59.507136 26489 ts_tablet_manager.cc:1428] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260927 22:14:59.575469 26192 tablet.cc:2369] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:59.593822 26494 raft_consensus.cc:482] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:59.594115 26494 raft_consensus.cc:504] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.595677 26494 leader_election.cc:288] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291)
I20260927 22:14:59.602433 26495 raft_consensus.cc:482] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:59.602751 26495 raft_consensus.cc:504] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.603284 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.596042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.23.173.67:47253)
0927 22:14:59.596269 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.596288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.596477 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.596989 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.596993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.597018 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.597206 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.597218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.598625 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.598631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.600268 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.600280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.602478 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.602484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.602675 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.602727 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.602817 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.602993 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":17}
I20260927 22:14:59.603319 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.596042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:47253 (local address 127.23.173.65:42895)
0927 22:14:59.596270 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.596278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.596381 (+ 103us) server_negotiation.cc:407] Connection header received
0927 22:14:59.596620 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.596626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.596756 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.596864 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.597366 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.598496 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.600425 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.601760 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.601933 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.602268 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.602969 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.603113 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:14:59.604086 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.596643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:32861 (local address 127.23.173.66:38291)
0927 22:14:59.596872 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.596878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.597232 (+ 354us) server_negotiation.cc:407] Connection header received
0927 22:14:59.597336 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.597341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.597424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.597497 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.598116 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.599052 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.601211 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.602642 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.602804 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.602874 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.603671 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.603808 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:14:59.604112 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.596384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38291 (local address 127.23.173.67:32861)
0927 22:14:59.597124 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.597138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.597235 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.597549 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.597553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.597577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.597931 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.597942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.599194 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.599198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.601028 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.601043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.602847 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.602853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.603180 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.603212 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.603633 (+ 421us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.603755 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":53,"thread_start_us":413,"threads_started":1}
I20260927 22:14:59.604264 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14" is_pre_election: true
I20260927 22:14:59.604772 26141 raft_consensus.cc:2454] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 0.
I20260927 22:14:59.605000 26495 leader_election.cc:288] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:14:59.605542 26350 leader_election.cc:302] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:14:59.605636 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969" is_pre_election: true
I20260927 22:14:59.606002 26494 raft_consensus.cc:2781] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:59.606154 26494 raft_consensus.cc:482] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:59.606256 26494 raft_consensus.cc:3037] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.606184 26285 raft_consensus.cc:2454] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 0.
W20260927 22:14:59.608875 26331 tablet.cc:2369] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:59.610399 26494 raft_consensus.cc:504] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.611122 26494 leader_election.cc:288] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 election: Requested vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291)
I20260927 22:14:59.611555 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14"
I20260927 22:14:59.611552 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969"
I20260927 22:14:59.611734 26141 raft_consensus.cc:3037] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.611742 26285 raft_consensus.cc:3037] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.613648 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.605251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38291 (local address 127.23.173.65:59145)
0927 22:14:59.606376 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.606395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.606575 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.607019 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.607023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.607052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.607462 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.607477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.608730 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.608739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.610980 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.610998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.612807 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.612813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.613035 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.613070 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.613168 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.613298 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":792,"threads_started":1}
I20260927 22:14:59.613682 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.605197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:59145 (local address 127.23.173.66:38291)
0927 22:14:59.605371 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.605379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.606475 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 22:14:59.606718 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.606722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.606787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.606842 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.607625 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.608585 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.611174 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.612367 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.612502 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.612570 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.613227 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.613374 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:14:59.615027 26285 raft_consensus.cc:2454] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 1.
I20260927 22:14:59.614885 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969" is_pre_election: true
I20260927 22:14:59.615271 26141 raft_consensus.cc:2454] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 1.
I20260927 22:14:59.615324 26284 raft_consensus.cc:2379] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd154cd665be48f5b701775fed332f14 in current term 1: Already voted for candidate 71f9c41a406e41e0af6f7120dae2936f in this term.
I20260927 22:14:59.615495 26350 leader_election.cc:302] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, a6aecc557fca4b7e9c283ed937977969; no voters:
I20260927 22:14:59.615805 26494 raft_consensus.cc:2781] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:59.616623 26494 raft_consensus.cc:686] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 1 LEADER]: Becoming Leader. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Running, Role: LEADER
I20260927 22:14:59.617138 26494 consensus_queue.cc:237] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
W20260927 22:14:59.618162 26470 tablet.cc:2369] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:14:59.619963 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.607240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:39811 (local address 127.23.173.67:37359)
0927 22:14:59.607455 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.607460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.610287 (+ 2827us) server_negotiation.cc:407] Connection header received
0927 22:14:59.610434 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.610440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.610504 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.610594 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:14:59.614388 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.615461 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.617569 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.618887 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.619062 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.619109 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.619569 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.619697 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:14:59.619984 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.606492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37359 (local address 127.23.173.65:39811)
0927 22:14:59.610204 (+ 3712us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.610221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.610325 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.610666 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.610670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.610693 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:14:59.611089 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.611100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.615759 (+ 4659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.615764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.617399 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.617430 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.619119 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.619129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.619339 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.619371 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.619523 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.619652 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"thread_start_us":3410,"threads_started":1}
I20260927 22:14:59.621107 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e01a3ba39dc4707938a56860b8b6be5" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f" is_pre_election: true
I20260927 22:14:59.622056 26071 leader_election.cc:302] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd154cd665be48f5b701775fed332f14; no voters: 71f9c41a406e41e0af6f7120dae2936f, a6aecc557fca4b7e9c283ed937977969
I20260927 22:14:59.622516 26495 raft_consensus.cc:2726] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:14:59.622154 25983 catalog_manager.cc:5495] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67). New cstate: current_term: 1 leader_uuid: "71f9c41a406e41e0af6f7120dae2936f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } health_report { overall_health: HEALTHY } } }
Sep 27 22:14:59 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547297, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:14:59.691028 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.676203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.23.173.126:35409)
0927 22:14:59.676406 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.676431 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.677055 (+ 624us) server_negotiation.cc:407] Connection header received
0927 22:14:59.677192 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.677196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.677274 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.677356 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:14:59.678510 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.679461 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.681097 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.681538 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.681632 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.685495 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:14:59.685499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:14:59.685502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:14:59.685530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:14:59.688205 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.689103 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.689106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.689108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.689236 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:14:59.689612 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:14:59.689615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:14:59.689617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:14:59.689875 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:14:59.690046 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.690475 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.690788 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":61}
I20260927 22:14:59.691150 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.676306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35409 (local address 127.0.0.1:46544)
0927 22:14:59.676948 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.676965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.677092 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.677409 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.677412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.678102 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:14:59.678386 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.678396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.679584 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.679588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.680829 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.680841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.681715 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.681720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.681889 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.682766 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:14:59.682783 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:14:59.685305 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:14:59.688328 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.688340 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.688349 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.688996 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.689334 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:14:59.689340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:14:59.689344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:14:59.689517 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:14:59.690027 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:14:59.690035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:14:59.690215 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.690396 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.690667 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":362,"threads_started":1}
I20260927 22:14:59.696874 25983 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:46544:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:14:59.698163 25983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:14:59.705298 26121 tablet_service.cc:1467] Processing CreateTablet for tablet 2b51975cb3084f55bda0ab01555d06b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77298e1aa674877990a82dfbc6d3238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:59.705438 26260 tablet_service.cc:1467] Processing CreateTablet for tablet 2b51975cb3084f55bda0ab01555d06b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77298e1aa674877990a82dfbc6d3238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:59.706344 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b51975cb3084f55bda0ab01555d06b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.706349 26121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b51975cb3084f55bda0ab01555d06b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.706629 26401 tablet_service.cc:1467] Processing CreateTablet for tablet 2b51975cb3084f55bda0ab01555d06b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77298e1aa674877990a82dfbc6d3238]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:14:59.707300 26401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b51975cb3084f55bda0ab01555d06b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:14:59.713035 26490 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: Bootstrap starting.
I20260927 22:14:59.713426 26488 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: Bootstrap starting.
I20260927 22:14:59.713425 26489 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: Bootstrap starting.
I20260927 22:14:59.714829 26490 tablet_bootstrap.cc:654] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.715051 26489 tablet_bootstrap.cc:654] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.715058 26488 tablet_bootstrap.cc:654] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: Neither blocks nor log segments found. Creating new log.
I20260927 22:14:59.717255 26490 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: No bootstrap required, opened a new log
I20260927 22:14:59.717411 26490 ts_tablet_manager.cc:1397] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:14:59.718358 26488 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: No bootstrap required, opened a new log
I20260927 22:14:59.718314 26490 raft_consensus.cc:348] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.718503 26488 ts_tablet_manager.cc:1397] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:14:59.718555 26490 raft_consensus.cc:374] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.718653 26490 raft_consensus.cc:729] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.718751 26489 tablet_bootstrap.cc:492] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: No bootstrap required, opened a new log
I20260927 22:14:59.718883 26489 ts_tablet_manager.cc:1397] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 22:14:59.718942 26490 consensus_queue.cc:260] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.719220 26488 raft_consensus.cc:348] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.719458 26488 raft_consensus.cc:374] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.719367 26494 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.719549 26488 raft_consensus.cc:729] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.719657 26494 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.719651 26489 raft_consensus.cc:348] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.719893 26489 raft_consensus.cc:374] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:14:59.719825 26488 consensus_queue.cc:260] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.719987 26489 raft_consensus.cc:729] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd154cd665be48f5b701775fed332f14, State: Initialized, Role: FOLLOWER
I20260927 22:14:59.720160 26490 ts_tablet_manager.cc:1428] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:14:59.720238 26489 consensus_queue.cc:260] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.720432 26488 ts_tablet_manager.cc:1428] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:14:59.720528 26496 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.720779 26496 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.720810 26489 ts_tablet_manager.cc:1428] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:14:59.720664 26495 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.720932 26495 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.829643 26495 raft_consensus.cc:482] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:14:59.829870 26495 raft_consensus.cc:504] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.830715 26495 leader_election.cc:288] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:14:59.831264 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b51975cb3084f55bda0ab01555d06b8" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969" is_pre_election: true
I20260927 22:14:59.831527 26285 raft_consensus.cc:2454] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 0.
I20260927 22:14:59.831626 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b51975cb3084f55bda0ab01555d06b8" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f" is_pre_election: true
I20260927 22:14:59.831988 26071 leader_election.cc:302] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6aecc557fca4b7e9c283ed937977969, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:14:59.831991 26424 raft_consensus.cc:2454] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 0.
I20260927 22:14:59.832245 26495 raft_consensus.cc:2781] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:14:59.832389 26495 raft_consensus.cc:482] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:14:59.832490 26495 raft_consensus.cc:3037] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.834714 26495 raft_consensus.cc:504] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.835237 26495 leader_election.cc:288] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 election: Requested vote from peers a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:14:59.835618 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b51975cb3084f55bda0ab01555d06b8" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969"
I20260927 22:14:59.835716 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b51975cb3084f55bda0ab01555d06b8" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f"
I20260927 22:14:59.835834 26285 raft_consensus.cc:3037] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.835952 26424 raft_consensus.cc:3037] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:14:59.838177 26424 raft_consensus.cc:2454] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 1.
I20260927 22:14:59.838274 26285 raft_consensus.cc:2454] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 1.
I20260927 22:14:59.838575 26071 leader_election.cc:302] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:14:59.838871 26495 raft_consensus.cc:2781] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:14:59.839350 26495 raft_consensus.cc:686] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [term 1 LEADER]: Becoming Leader. State: Replica: bd154cd665be48f5b701775fed332f14, State: Running, Role: LEADER
I20260927 22:14:59.839919 26495 consensus_queue.cc:237] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:14:59.843325 26511 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: TxnStatusTablet state changed. Reason: New leader bd154cd665be48f5b701775fed332f14. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.843750 26511 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:59.844412 26513 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:14:59.844666 26513 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:14:59.844686 25982 catalog_manager.cc:5495] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd154cd665be48f5b701775fed332f14 (127.23.173.65). New cstate: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260927 22:14:59.873615 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.868138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.0.0.1:33208)
0927 22:14:59.868362 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.868384 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.868528 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.869014 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.869019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.869030 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:14:59.869226 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.869235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.870330 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.870338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.871313 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.871324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.872096 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.872102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.872258 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.872311 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.873081 (+ 770us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.873101 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.873171 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.873314 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:14:59.873632 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.868475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.23.173.65:42895)
0927 22:14:59.868666 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.868672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.868691 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:14:59.868772 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.868776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.868857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.869004 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:59.869361 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.870203 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.871458 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.871936 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.872063 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.872499 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.872973 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.873128 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.873188 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.873264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:14:59.880829 26424 raft_consensus.cc:1261] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Refusing update from remote peer bd154cd665be48f5b701775fed332f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:59.880975 26285 raft_consensus.cc:1261] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Refusing update from remote peer bd154cd665be48f5b701775fed332f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:59.881647 26511 consensus_queue.cc:1035] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:59.882072 26495 consensus_queue.cc:1035] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:59.895804 26494 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: TxnStatusTablet state changed. Reason: New leader bd154cd665be48f5b701775fed332f14. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.896121 26494 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.898159 26496 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: TxnStatusTablet state changed. Reason: New leader bd154cd665be48f5b701775fed332f14. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.898433 26496 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.899353 26516 mvcc.cc:205] Tried to move back new op lower bound from 7334081740303622144 to 7334081740148416512. Current Snapshot: MvccSnapshot[applied={T|T < 7334081740303622144}]
I20260927 22:14:59.899977 26517 mvcc.cc:205] Tried to move back new op lower bound from 7334081740303622144 to 7334081740148416512. Current Snapshot: MvccSnapshot[applied={T|T < 7334081740303622144}]
I20260927 22:14:59.901611 26495 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.901930 26495 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:59.902149 26519 mvcc.cc:205] Tried to move back new op lower bound from 7334081740303622144 to 7334081740148416512. Current Snapshot: MvccSnapshot[applied={T|T < 7334081740303622144}]
I20260927 22:14:59.902686 26495 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.902981 26495 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P bd154cd665be48f5b701775fed332f14: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:14:59.909397 26496 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.909659 26496 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P a6aecc557fca4b7e9c283ed937977969: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.912513 26494 tablet_replica.cc:439] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } } }
I20260927 22:14:59.912774 26494 tablet_replica.cc:442] T 2b51975cb3084f55bda0ab01555d06b8 P 71f9c41a406e41e0af6f7120dae2936f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:14:59.919091 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.913888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.23.173.67:37359)
0927 22:14:59.914066 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.914074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.914093 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:14:59.914356 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.914361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.914441 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.914585 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:59.914933 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.915720 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.917017 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.917419 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.917554 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.918027 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.918505 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.918654 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.918732 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.918810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:14:59.935393 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.929606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.23.173.67:34903)
0927 22:14:59.930226 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.930240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.930341 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.930685 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.930688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.930697 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:14:59.930956 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.930962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.932389 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.932392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.933378 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.933389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.934390 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.934394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.934506 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.934553 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.934974 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.934988 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.935047 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.935138 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":391,"threads_started":1}
I20260927 22:14:59.935477 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.929775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34903 (local address 127.23.173.65:42895)
0927 22:14:59.930002 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.930009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.930301 (+ 292us) server_negotiation.cc:407] Connection header received
0927 22:14:59.930461 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.930471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.930569 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.930657 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:59.931091 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.932264 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.933505 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.934198 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.934360 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.934745 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.934879 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.934971 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.935103 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.935183 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:14:59.945142 26284 raft_consensus.cc:1261] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Refusing update from remote peer 71f9c41a406e41e0af6f7120dae2936f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:59.945565 26141 raft_consensus.cc:1261] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Refusing update from remote peer 71f9c41a406e41e0af6f7120dae2936f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:14:59.945797 26500 consensus_queue.cc:1035] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:59.946220 26494 consensus_queue.cc:1035] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:14:59.983893 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.976408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:34735 (local address 127.23.173.67:37359)
0927 22:14:59.976587 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:14:59.976593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:14:59.977686 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 22:14:59.977887 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:14:59.977893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:14:59.978023 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:14:59.978126 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:14:59.978656 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.979644 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.981902 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.982490 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.982635 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.983138 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.983264 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.983376 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:14:59.983529 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:14:59.983614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:14:59.984069 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:14:59.976496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37359 (local address 127.23.173.65:34735)
0927 22:14:59.977587 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0927 22:14:59.977603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:14:59.977759 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:14:59.978146 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:14:59.978151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:14:59.978163 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:14:59.978499 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.978507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.979800 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.979806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.981751 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:14:59.981767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:14:59.982694 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:14:59.982700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:14:59.982881 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:14:59.982951 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:14:59.983377 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:14:59.983397 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:14:59.983499 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:14:59.983644 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":703,"threads_started":1}
I20260927 22:15:00.026544 25981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:00.026836 25981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:00.030691 25981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Sending DeleteTablet for 3 replicas of tablet 3e01a3ba39dc4707938a56860b8b6be5
I20260927 22:15:00.032198 26260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35810
I20260927 22:15:00.032766 25981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:15:00.032882 26401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:54156
W20260927 22:15:00.033340 25981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:00.036235 26121 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e01a3ba39dc4707938a56860b8b6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33202
I20260927 22:15:00.041517 26259 tablet_service.cc:1467] Processing CreateTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e (DEFAULT_TABLE table=test-table [id=3373d87f4382449398d70375c17a6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:00.041669 26398 tablet_service.cc:1467] Processing CreateTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e (DEFAULT_TABLE table=test-table [id=3373d87f4382449398d70375c17a6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:00.042073 26259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d185f3d3543b6ab40e5e7eaf50b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:00.042245 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d185f3d3543b6ab40e5e7eaf50b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:00.044279 26538 tablet_replica.cc:330] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: stopping tablet replica
I20260927 22:15:00.044745 26538 raft_consensus.cc:2229] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:00.045081 26538 raft_consensus.cc:2258] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:00.046103 26120 tablet_service.cc:1467] Processing CreateTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e (DEFAULT_TABLE table=test-table [id=3373d87f4382449398d70375c17a6309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:00.046605 26120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d185f3d3543b6ab40e5e7eaf50b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:00.049692 26490 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Bootstrap starting.
I20260927 22:15:00.056166 26490 tablet_bootstrap.cc:654] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:00.060839 26488 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Bootstrap starting.
I20260927 22:15:00.062536 26488 tablet_bootstrap.cc:654] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:00.065786 26488 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: No bootstrap required, opened a new log
I20260927 22:15:00.065989 26488 ts_tablet_manager.cc:1397] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:15:00.066737 26488 raft_consensus.cc:348] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.066993 26488 raft_consensus.cc:374] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:00.067070 26488 raft_consensus.cc:729] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Initialized, Role: FOLLOWER
I20260927 22:15:00.067322 26488 consensus_queue.cc:260] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.068925 26488 ts_tablet_manager.cc:1428] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:15:00.075584 26489 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Bootstrap starting.
I20260927 22:15:00.077701 26489 tablet_bootstrap.cc:654] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:00.077898 26490 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: No bootstrap required, opened a new log
I20260927 22:15:00.078122 26490 ts_tablet_manager.cc:1397] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.002s
I20260927 22:15:00.078945 26490 raft_consensus.cc:348] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.079198 26490 raft_consensus.cc:374] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:00.079304 26490 raft_consensus.cc:729] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Initialized, Role: FOLLOWER
I20260927 22:15:00.085317 26538 ts_tablet_manager.cc:1905] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:00.088280 26489 tablet_bootstrap.cc:492] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: No bootstrap required, opened a new log
I20260927 22:15:00.088515 26489 ts_tablet_manager.cc:1397] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260927 22:15:00.089386 26489 raft_consensus.cc:348] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.089702 26489 raft_consensus.cc:374] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:00.089851 26489 raft_consensus.cc:729] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd154cd665be48f5b701775fed332f14, State: Initialized, Role: FOLLOWER
I20260927 22:15:00.090274 26489 consensus_queue.cc:260] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.090759 26490 consensus_queue.cc:260] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.091379 26490 ts_tablet_manager.cc:1428] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20260927 22:15:00.095012 26537 tablet_replica.cc:330] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: stopping tablet replica
I20260927 22:15:00.095209 26489 ts_tablet_manager.cc:1428] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 22:15:00.095475 26537 raft_consensus.cc:2229] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:00.096135 26537 raft_consensus.cc:2258] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:00.096958 26536 tablet_replica.cc:330] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: stopping tablet replica
I20260927 22:15:00.097380 26536 raft_consensus.cc:2229] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:00.097738 26536 raft_consensus.cc:2258] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:00.100113 26537 ts_tablet_manager.cc:1905] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:00.103022 26538 ts_tablet_manager.cc:1918] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:00.103236 26538 log.cc:1196] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/3e01a3ba39dc4707938a56860b8b6be5
I20260927 22:15:00.103889 26538 ts_tablet_manager.cc:1939] T 3e01a3ba39dc4707938a56860b8b6be5 P bd154cd665be48f5b701775fed332f14: Deleting consensus metadata
I20260927 22:15:00.104202 26536 ts_tablet_manager.cc:1905] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:00.105679 25968 catalog_manager.cc:4841] TS bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895): tablet 3e01a3ba39dc4707938a56860b8b6be5 (table test-table [id=d724cfdac2da4747bf3824574f02892e]) successfully deleted
I20260927 22:15:00.108572 26537 ts_tablet_manager.cc:1918] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:00.108753 26537 log.cc:1196] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/3e01a3ba39dc4707938a56860b8b6be5
I20260927 22:15:00.109318 26537 ts_tablet_manager.cc:1939] T 3e01a3ba39dc4707938a56860b8b6be5 P 71f9c41a406e41e0af6f7120dae2936f: Deleting consensus metadata
I20260927 22:15:00.110939 25968 catalog_manager.cc:4841] TS 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359): tablet 3e01a3ba39dc4707938a56860b8b6be5 (table test-table [id=d724cfdac2da4747bf3824574f02892e]) successfully deleted
I20260927 22:15:00.115224 26536 ts_tablet_manager.cc:1918] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:00.115406 26536 log.cc:1196] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/3e01a3ba39dc4707938a56860b8b6be5
I20260927 22:15:00.115808 26536 ts_tablet_manager.cc:1939] T 3e01a3ba39dc4707938a56860b8b6be5 P a6aecc557fca4b7e9c283ed937977969: Deleting consensus metadata
I20260927 22:15:00.117106 25968 catalog_manager.cc:4841] TS a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291): tablet 3e01a3ba39dc4707938a56860b8b6be5 (table test-table [id=d724cfdac2da4747bf3824574f02892e]) successfully deleted
I20260927 22:15:00.231106 26524 raft_consensus.cc:482] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:00.231333 26524 raft_consensus.cc:504] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.232089 26524 leader_election.cc:288] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:15:00.232697 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d185f3d3543b6ab40e5e7eaf50b8e" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969" is_pre_election: true
I20260927 22:15:00.232702 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d185f3d3543b6ab40e5e7eaf50b8e" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f" is_pre_election: true
I20260927 22:15:00.232968 26284 raft_consensus.cc:2454] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 0.
I20260927 22:15:00.233011 26424 raft_consensus.cc:2454] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 0.
I20260927 22:15:00.233531 26071 leader_election.cc:302] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:15:00.233816 26524 raft_consensus.cc:2781] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:00.233946 26524 raft_consensus.cc:482] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:00.234063 26524 raft_consensus.cc:3037] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:00.236176 26524 raft_consensus.cc:504] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.236794 26524 leader_election.cc:288] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 election: Requested vote from peers a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:15:00.237294 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d185f3d3543b6ab40e5e7eaf50b8e" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969"
I20260927 22:15:00.237294 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d185f3d3543b6ab40e5e7eaf50b8e" candidate_uuid: "bd154cd665be48f5b701775fed332f14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f"
I20260927 22:15:00.237509 26284 raft_consensus.cc:3037] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:00.237514 26424 raft_consensus.cc:3037] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:00.240451 26284 raft_consensus.cc:2454] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 1.
I20260927 22:15:00.240480 26424 raft_consensus.cc:2454] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd154cd665be48f5b701775fed332f14 in term 1.
I20260927 22:15:00.240885 26071 leader_election.cc:302] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6aecc557fca4b7e9c283ed937977969, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:15:00.241215 26524 raft_consensus.cc:2781] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:00.241425 26524 raft_consensus.cc:686] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 1 LEADER]: Becoming Leader. State: Replica: bd154cd665be48f5b701775fed332f14, State: Running, Role: LEADER
I20260927 22:15:00.241748 26524 consensus_queue.cc:237] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:00.244899 25982 catalog_manager.cc:5495] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd154cd665be48f5b701775fed332f14 (127.23.173.65). New cstate: current_term: 1 leader_uuid: "bd154cd665be48f5b701775fed332f14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:00.271433 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:00.266160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.23.173.65:42895)
0927 22:15:00.266323 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:00.266329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:00.266346 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:15:00.266403 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:00.266406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:00.266463 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:00.266575 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:00.266942 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.267744 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.269034 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.269520 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.269656 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:00.270447 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:00.270567 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:00.270671 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:00.271108 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 22:15:00.271204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:15:00.284845 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:00.279462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:42941 (local address 127.23.173.65:42895)
0927 22:15:00.279622 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:00.279628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:00.279734 (+ 106us) server_negotiation.cc:407] Connection header received
0927 22:15:00.279918 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:00.279923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:00.279992 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:00.280079 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:00.280397 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.281200 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.282988 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.283499 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.283636 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:00.284072 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:00.284171 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:00.284271 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:00.284403 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:15:00.284488 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":46}
I20260927 22:15:00.284857 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:00.279421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.23.173.65:42941)
0927 22:15:00.279642 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:00.279661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:00.279806 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:00.280090 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:00.280095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:00.280104 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:00.280280 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:00.280288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.281446 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.281451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:00.282851 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:00.282863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:00.283643 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:00.283648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:00.283890 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:00.283922 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:00.284271 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:00.284285 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:15:00.284385 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:15:00.284531 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":26}
I20260927 22:15:00.293589 26424 raft_consensus.cc:1261] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Refusing update from remote peer bd154cd665be48f5b701775fed332f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:00.293702 26285 raft_consensus.cc:1261] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Refusing update from remote peer bd154cd665be48f5b701775fed332f14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:00.294340 26495 consensus_queue.cc:1035] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:00.294603 26511 consensus_queue.cc:1035] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:15:00.326656 26516 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:15:01.933916 25982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:01.934209 25982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:01.938143 25982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Sending DeleteTablet for 3 replicas of tablet a56d185f3d3543b6ab40e5e7eaf50b8e
I20260927 22:15:01.938819 26120 tablet_service.cc:1514] Processing DeleteTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:01 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33202
I20260927 22:15:01.939002 26260 tablet_service.cc:1514] Processing DeleteTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:01 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35810
I20260927 22:15:01.939040 26401 tablet_service.cc:1514] Processing DeleteTablet for tablet a56d185f3d3543b6ab40e5e7eaf50b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:01 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260927 22:15:01.951638 26570 tablet_replica.cc:330] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: stopping tablet replica
I20260927 22:15:01.951735 26569 tablet_replica.cc:330] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: stopping tablet replica
I20260927 22:15:01.952075 26569 raft_consensus.cc:2229] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:01.952100 26570 raft_consensus.cc:2229] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:01.952407 26569 raft_consensus.cc:2258] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:01.952579 26570 raft_consensus.cc:2258] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:01.953783 26570 ts_tablet_manager.cc:1905] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:01.953801 26569 ts_tablet_manager.cc:1905] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:01.960636 26568 tablet_replica.cc:330] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: stopping tablet replica
I20260927 22:15:01.960846 26568 raft_consensus.cc:2229] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:01.961421 26568 raft_consensus.cc:2258] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:01.961532 26570 ts_tablet_manager.cc:1918] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:01.961660 26570 log.cc:1196] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/a56d185f3d3543b6ab40e5e7eaf50b8e
I20260927 22:15:01.961838 26569 ts_tablet_manager.cc:1918] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:01.962002 26569 log.cc:1196] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/a56d185f3d3543b6ab40e5e7eaf50b8e
I20260927 22:15:01.962131 26570 ts_tablet_manager.cc:1939] T a56d185f3d3543b6ab40e5e7eaf50b8e P 71f9c41a406e41e0af6f7120dae2936f: Deleting consensus metadata
I20260927 22:15:01.962457 26569 ts_tablet_manager.cc:1939] T a56d185f3d3543b6ab40e5e7eaf50b8e P a6aecc557fca4b7e9c283ed937977969: Deleting consensus metadata
I20260927 22:15:01.963721 25968 catalog_manager.cc:4841] TS a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291): tablet a56d185f3d3543b6ab40e5e7eaf50b8e (table test-table [id=3373d87f4382449398d70375c17a6309]) successfully deleted
I20260927 22:15:01.964000 25968 catalog_manager.cc:4841] TS 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359): tablet a56d185f3d3543b6ab40e5e7eaf50b8e (table test-table [id=3373d87f4382449398d70375c17a6309]) successfully deleted
I20260927 22:15:01.964443 26568 ts_tablet_manager.cc:1905] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:01.969440 26568 ts_tablet_manager.cc:1918] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:01.969580 26568 log.cc:1196] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/a56d185f3d3543b6ab40e5e7eaf50b8e
I20260927 22:15:01.970028 26568 ts_tablet_manager.cc:1939] T a56d185f3d3543b6ab40e5e7eaf50b8e P bd154cd665be48f5b701775fed332f14: Deleting consensus metadata
I20260927 22:15:01.970997 25968 catalog_manager.cc:4841] TS bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895): tablet a56d185f3d3543b6ab40e5e7eaf50b8e (table test-table [id=3373d87f4382449398d70375c17a6309]) successfully deleted
Sep 27 22:15:01 dist-test-slave-3wc3 krb5kdc[25939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:15:01 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:01.999293 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:01.984996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.23.173.126:35409)
0927 22:15:01.985598 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:01.985607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:01.985633 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:15:01.985859 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:01.985863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:01.985926 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:01.986120 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:01.987422 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:01.988187 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:01.989428 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:01.989856 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:01.990036 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:01.994045 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:01.994052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:01.994055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:01.994090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:01.996696 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:01.997341 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:01.997345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:01.997348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:01.997489 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:01.997885 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:01.997888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:01.997890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:01.998211 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:01.998410 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:01.998662 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:15:01.998919 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
I20260927 22:15:02.002960 25982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:02.003772 25982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:02.010494 26120 tablet_service.cc:1467] Processing CreateTablet for tablet a1df598e2cb1462b836feef43b9ecac0 (DEFAULT_TABLE table=test-table [id=24088d5f24234f7d991f986ef164e0d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.010578 26260 tablet_service.cc:1467] Processing CreateTablet for tablet a1df598e2cb1462b836feef43b9ecac0 (DEFAULT_TABLE table=test-table [id=24088d5f24234f7d991f986ef164e0d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.010979 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1df598e2cb1462b836feef43b9ecac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.010994 26120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1df598e2cb1462b836feef43b9ecac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.015949 26579 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Bootstrap starting.
I20260927 22:15:02.016320 26401 tablet_service.cc:1467] Processing CreateTablet for tablet a1df598e2cb1462b836feef43b9ecac0 (DEFAULT_TABLE table=test-table [id=24088d5f24234f7d991f986ef164e0d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.016789 26401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1df598e2cb1462b836feef43b9ecac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.017407 26580 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Bootstrap starting.
I20260927 22:15:02.017925 26579 tablet_bootstrap.cc:654] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.018941 26580 tablet_bootstrap.cc:654] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.020604 26579 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: No bootstrap required, opened a new log
I20260927 22:15:02.020758 26579 ts_tablet_manager.cc:1397] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:15:02.021423 26580 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: No bootstrap required, opened a new log
I20260927 22:15:02.021564 26580 ts_tablet_manager.cc:1397] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 22:15:02.021624 26579 raft_consensus.cc:348] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.021895 26579 raft_consensus.cc:374] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.022013 26579 raft_consensus.cc:729] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.022254 26579 consensus_queue.cc:260] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.022421 26580 raft_consensus.cc:348] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.022677 26580 raft_consensus.cc:374] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.022812 26580 raft_consensus.cc:729] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd154cd665be48f5b701775fed332f14, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.023229 26579 ts_tablet_manager.cc:1428] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:15:02.023088 26580 consensus_queue.cc:260] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.024024 26584 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Bootstrap starting.
I20260927 22:15:02.025277 26580 ts_tablet_manager.cc:1428] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:15:02.025671 26584 tablet_bootstrap.cc:654] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.028034 26584 tablet_bootstrap.cc:492] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: No bootstrap required, opened a new log
I20260927 22:15:02.028232 26584 ts_tablet_manager.cc:1397] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 22:15:02.029044 26584 raft_consensus.cc:348] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.029316 26584 raft_consensus.cc:374] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.029421 26584 raft_consensus.cc:729] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.029651 26584 consensus_queue.cc:260] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.030409 26584 ts_tablet_manager.cc:1428] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:15:02.242443 26586 raft_consensus.cc:482] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:02.242691 26586 raft_consensus.cc:504] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.243392 26586 leader_election.cc:288] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291)
I20260927 22:15:02.244032 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1df598e2cb1462b836feef43b9ecac0" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14" is_pre_election: true
I20260927 22:15:02.244066 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1df598e2cb1462b836feef43b9ecac0" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969" is_pre_election: true
I20260927 22:15:02.244307 26285 raft_consensus.cc:2454] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 0.
I20260927 22:15:02.244308 26141 raft_consensus.cc:2454] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 0.
I20260927 22:15:02.244803 26350 leader_election.cc:302] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:15:02.245059 26586 raft_consensus.cc:2781] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:02.245179 26586 raft_consensus.cc:482] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:02.245287 26586 raft_consensus.cc:3037] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.247680 26586 raft_consensus.cc:504] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.248301 26586 leader_election.cc:288] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 election: Requested vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291)
I20260927 22:15:02.248744 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1df598e2cb1462b836feef43b9ecac0" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14"
I20260927 22:15:02.248822 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1df598e2cb1462b836feef43b9ecac0" candidate_uuid: "71f9c41a406e41e0af6f7120dae2936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aecc557fca4b7e9c283ed937977969"
I20260927 22:15:02.248957 26141 raft_consensus.cc:3037] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.249054 26285 raft_consensus.cc:3037] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.251495 26141 raft_consensus.cc:2454] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 1.
I20260927 22:15:02.251529 26285 raft_consensus.cc:2454] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71f9c41a406e41e0af6f7120dae2936f in term 1.
I20260927 22:15:02.251993 26350 leader_election.cc:302] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, a6aecc557fca4b7e9c283ed937977969; no voters:
I20260927 22:15:02.252277 26586 raft_consensus.cc:2781] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:02.252696 26586 raft_consensus.cc:686] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 1 LEADER]: Becoming Leader. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Running, Role: LEADER
I20260927 22:15:02.253082 26586 consensus_queue.cc:237] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.255569 25982 catalog_manager.cc:5495] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67). New cstate: current_term: 1 leader_uuid: "71f9c41a406e41e0af6f7120dae2936f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } health_report { overall_health: HEALTHY } } }
I20260927 22:15:02.324769 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.318121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.23.173.67:37359)
0927 22:15:02.319294 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:02.319298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:02.319316 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:02.319365 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:02.319369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:02.319425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:02.319512 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:02.319911 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.320767 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.322561 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.323219 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.323339 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.323809 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:02.323967 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:02.324068 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:02.324294 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 22:15:02.324384 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":929,"threads_started":1}
I20260927 22:15:02.334883 26285 raft_consensus.cc:1261] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Refusing update from remote peer 71f9c41a406e41e0af6f7120dae2936f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:02.335129 26141 raft_consensus.cc:1261] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Refusing update from remote peer 71f9c41a406e41e0af6f7120dae2936f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:02.335408 26586 consensus_queue.cc:1035] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:02.335765 26587 consensus_queue.cc:1035] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:02.392055 25984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:02.392344 25984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:02.395480 25984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Sending DeleteTablet for 3 replicas of tablet a1df598e2cb1462b836feef43b9ecac0
I20260927 22:15:02.396310 26120 tablet_service.cc:1514] Processing DeleteTablet for tablet a1df598e2cb1462b836feef43b9ecac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:02 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33202
I20260927 22:15:02.396361 26401 tablet_service.cc:1514] Processing DeleteTablet for tablet a1df598e2cb1462b836feef43b9ecac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:02 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260927 22:15:02.396577 26568 tablet_replica.cc:330] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: stopping tablet replica
I20260927 22:15:02.396620 26570 tablet_replica.cc:330] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: stopping tablet replica
I20260927 22:15:02.396777 26568 raft_consensus.cc:2229] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:02.396822 26570 raft_consensus.cc:2229] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:02.397008 26568 raft_consensus.cc:2258] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:02.397060 26260 tablet_service.cc:1514] Processing DeleteTablet for tablet a1df598e2cb1462b836feef43b9ecac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:02 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35810
I20260927 22:15:02.397145 26570 raft_consensus.cc:2258] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:02.397432 26569 tablet_replica.cc:330] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: stopping tablet replica
I20260927 22:15:02.397303 25984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:15:02.397643 26569 raft_consensus.cc:2229] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:02.397822 26569 raft_consensus.cc:2258] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 22:15:02.397989 25984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:02.398357 26570 ts_tablet_manager.cc:1905] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:02.399036 26568 ts_tablet_manager.cc:1905] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:02.402554 26569 ts_tablet_manager.cc:1905] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:02.404896 26570 ts_tablet_manager.cc:1918] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:02.405064 26570 log.cc:1196] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/a1df598e2cb1462b836feef43b9ecac0
I20260927 22:15:02.405400 26401 tablet_service.cc:1467] Processing CreateTablet for tablet 0219f2942dfe4e3b9d90690175572b68 (DEFAULT_TABLE table=test-table [id=7efdbc84e12d4583be593a83ceaef047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.405542 26570 ts_tablet_manager.cc:1939] T a1df598e2cb1462b836feef43b9ecac0 P 71f9c41a406e41e0af6f7120dae2936f: Deleting consensus metadata
I20260927 22:15:02.405822 26401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0219f2942dfe4e3b9d90690175572b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.406579 25968 catalog_manager.cc:4841] TS 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359): tablet a1df598e2cb1462b836feef43b9ecac0 (table test-table [id=24088d5f24234f7d991f986ef164e0d8]) successfully deleted
I20260927 22:15:02.407037 26260 tablet_service.cc:1467] Processing CreateTablet for tablet 0219f2942dfe4e3b9d90690175572b68 (DEFAULT_TABLE table=test-table [id=7efdbc84e12d4583be593a83ceaef047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.407473 26260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0219f2942dfe4e3b9d90690175572b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.408969 26568 ts_tablet_manager.cc:1918] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:02.409129 26568 log.cc:1196] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/a1df598e2cb1462b836feef43b9ecac0
I20260927 22:15:02.409546 26568 ts_tablet_manager.cc:1939] T a1df598e2cb1462b836feef43b9ecac0 P bd154cd665be48f5b701775fed332f14: Deleting consensus metadata
I20260927 22:15:02.410584 26569 ts_tablet_manager.cc:1918] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:02.410740 26569 log.cc:1196] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/a1df598e2cb1462b836feef43b9ecac0
I20260927 22:15:02.411198 26569 ts_tablet_manager.cc:1939] T a1df598e2cb1462b836feef43b9ecac0 P a6aecc557fca4b7e9c283ed937977969: Deleting consensus metadata
I20260927 22:15:02.412209 25968 catalog_manager.cc:4841] TS bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895): tablet a1df598e2cb1462b836feef43b9ecac0 (table test-table [id=24088d5f24234f7d991f986ef164e0d8]) successfully deleted
I20260927 22:15:02.412472 25968 catalog_manager.cc:4841] TS a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291): tablet a1df598e2cb1462b836feef43b9ecac0 (table test-table [id=24088d5f24234f7d991f986ef164e0d8]) successfully deleted
I20260927 22:15:02.412648 26120 tablet_service.cc:1467] Processing CreateTablet for tablet 0219f2942dfe4e3b9d90690175572b68 (DEFAULT_TABLE table=test-table [id=7efdbc84e12d4583be593a83ceaef047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:02.413071 26120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0219f2942dfe4e3b9d90690175572b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:02.414398 26584 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Bootstrap starting.
I20260927 22:15:02.415525 26579 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Bootstrap starting.
I20260927 22:15:02.416013 26584 tablet_bootstrap.cc:654] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.417213 26579 tablet_bootstrap.cc:654] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.418406 26584 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: No bootstrap required, opened a new log
I20260927 22:15:02.418586 26584 ts_tablet_manager.cc:1397] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260927 22:15:02.419178 26580 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Bootstrap starting.
I20260927 22:15:02.419389 26584 raft_consensus.cc:348] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.419618 26579 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: No bootstrap required, opened a new log
I20260927 22:15:02.419648 26584 raft_consensus.cc:374] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.419727 26584 raft_consensus.cc:729] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f9c41a406e41e0af6f7120dae2936f, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.419780 26579 ts_tablet_manager.cc:1397] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 22:15:02.420049 26584 consensus_queue.cc:260] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.420509 26579 raft_consensus.cc:348] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.420711 26584 ts_tablet_manager.cc:1428] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:02.420764 26579 raft_consensus.cc:374] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.420836 26579 raft_consensus.cc:729] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.420935 26580 tablet_bootstrap.cc:654] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:02.421103 26579 consensus_queue.cc:260] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.421615 26579 ts_tablet_manager.cc:1428] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:15:02.423022 26580 tablet_bootstrap.cc:492] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: No bootstrap required, opened a new log
I20260927 22:15:02.423396 26580 ts_tablet_manager.cc:1397] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260927 22:15:02.424180 26580 raft_consensus.cc:348] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.424538 26580 raft_consensus.cc:374] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:02.424623 26580 raft_consensus.cc:729] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd154cd665be48f5b701775fed332f14, State: Initialized, Role: FOLLOWER
I20260927 22:15:02.424938 26580 consensus_queue.cc:260] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.425513 26580 ts_tablet_manager.cc:1428] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:15:02.892132 26583 raft_consensus.cc:482] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:02.892333 26583 raft_consensus.cc:504] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.893730 26583 leader_election.cc:288] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:15:02.900506 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.894188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:55877 (local address 127.23.173.65:42895)
0927 22:15:02.894628 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:02.894633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:02.894732 (+ 99us) server_negotiation.cc:407] Connection header received
0927 22:15:02.894951 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:02.894957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:02.895034 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:02.895731 (+ 697us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:02.895856 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.896582 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.898503 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.899495 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.899625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.899676 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:02.899989 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:15:02.900134 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":262,"threads_started":1}
I20260927 22:15:02.900529 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.894133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.23.173.66:55877)
0927 22:15:02.894610 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:02.894631 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:02.894774 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:02.895250 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:02.895259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:02.895305 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:02.895613 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:02.895622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.896715 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.896721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:02.898340 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:02.898351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.899648 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.899654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:02.899835 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.899873 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:15:02.899972 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:15:02.900128 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":251,"threads_started":1}
I20260927 22:15:02.901172 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.894787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37359 (local address 127.23.173.66:53865)
0927 22:15:02.895339 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:02.895366 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:02.895481 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:02.895795 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:02.895799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:02.895821 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:02.896129 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:02.896137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.897319 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.897323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:02.898936 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:02.898952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.900392 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.900398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:02.900582 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.900619 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:15:02.900709 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:15:02.900838 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":209,"threads_started":1}
I20260927 22:15:02.901206 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.894973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:53865 (local address 127.23.173.67:37359)
0927 22:15:02.895497 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:02.895504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:02.895523 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:15:02.895598 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:02.895603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:02.895673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:02.895778 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:02.896274 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.897183 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.899119 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.900158 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.900309 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.900382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:02.900805 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:15:02.900949 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":324,"threads_started":1}
I20260927 22:15:02.901201 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0219f2942dfe4e3b9d90690175572b68" candidate_uuid: "a6aecc557fca4b7e9c283ed937977969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14" is_pre_election: true
I20260927 22:15:02.901496 26141 raft_consensus.cc:2454] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aecc557fca4b7e9c283ed937977969 in term 0.
I20260927 22:15:02.901820 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0219f2942dfe4e3b9d90690175572b68" candidate_uuid: "a6aecc557fca4b7e9c283ed937977969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f" is_pre_election: true
I20260927 22:15:02.902104 26424 raft_consensus.cc:2454] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aecc557fca4b7e9c283ed937977969 in term 0.
I20260927 22:15:02.902060 26211 leader_election.cc:302] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6aecc557fca4b7e9c283ed937977969, bd154cd665be48f5b701775fed332f14; no voters:
I20260927 22:15:02.902463 26583 raft_consensus.cc:2781] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:02.902587 26583 raft_consensus.cc:482] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:02.902702 26583 raft_consensus.cc:3037] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.905043 26583 raft_consensus.cc:504] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.905634 26583 leader_election.cc:288] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [CANDIDATE]: Term 1 election: Requested vote from peers bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895), 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359)
I20260927 22:15:02.906121 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0219f2942dfe4e3b9d90690175572b68" candidate_uuid: "a6aecc557fca4b7e9c283ed937977969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd154cd665be48f5b701775fed332f14"
I20260927 22:15:02.906219 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0219f2942dfe4e3b9d90690175572b68" candidate_uuid: "a6aecc557fca4b7e9c283ed937977969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f9c41a406e41e0af6f7120dae2936f"
I20260927 22:15:02.906323 26141 raft_consensus.cc:3037] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.906477 26424 raft_consensus.cc:3037] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:02.908423 26424 raft_consensus.cc:2454] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aecc557fca4b7e9c283ed937977969 in term 1.
I20260927 22:15:02.908520 26141 raft_consensus.cc:2454] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aecc557fca4b7e9c283ed937977969 in term 1.
I20260927 22:15:02.908839 26211 leader_election.cc:302] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71f9c41a406e41e0af6f7120dae2936f, a6aecc557fca4b7e9c283ed937977969; no voters:
I20260927 22:15:02.909129 26583 raft_consensus.cc:2781] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:02.909533 26583 raft_consensus.cc:686] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 1 LEADER]: Becoming Leader. State: Replica: a6aecc557fca4b7e9c283ed937977969, State: Running, Role: LEADER
I20260927 22:15:02.909986 26583 consensus_queue.cc:237] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } }
I20260927 22:15:02.913174 25984 catalog_manager.cc:5495] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6aecc557fca4b7e9c283ed937977969 (127.23.173.66). New cstate: current_term: 1 leader_uuid: "a6aecc557fca4b7e9c283ed937977969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aecc557fca4b7e9c283ed937977969" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:02.991180 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:02.985463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.23.173.66:38291)
0927 22:15:02.986006 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:02.986011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:02.986029 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:02.986216 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:02.986220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:02.986293 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:02.986471 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:02.986929 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.987679 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.988958 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:02.989359 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:02.989497 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:02.989917 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:02.990534 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:02.990692 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:02.990783 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:15:02.990863 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260927 22:15:03.005748 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:03.000735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:57267 (local address 127.23.173.65:42895)
0927 22:15:03.000918 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:03.000923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:03.001509 (+ 586us) server_negotiation.cc:407] Connection header received
0927 22:15:03.001578 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:03.001582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:03.001654 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:03.001735 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:03.002292 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.003011 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.004304 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.004706 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.004799 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:03.005179 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:03.005290 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:03.005365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:03.005481 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:15:03.005543 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:15:03.005843 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:03.000650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42895 (local address 127.23.173.66:57267)
0927 22:15:03.001289 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:03.001304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:03.001413 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:03.001775 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:03.001778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:03.001797 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:03.002170 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:03.002178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.003124 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.003128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:03.004179 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:03.004190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.004851 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.004855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:03.004982 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:03.005036 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:03.005379 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:03.005387 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:15:03.005446 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 22:15:03.005548 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":323,"threads_started":1}
I20260927 22:15:03.014304 26424 raft_consensus.cc:1261] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Refusing update from remote peer a6aecc557fca4b7e9c283ed937977969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:03.014292 26141 raft_consensus.cc:1261] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Refusing update from remote peer a6aecc557fca4b7e9c283ed937977969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:03.014988 26583 consensus_queue.cc:1035] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f9c41a406e41e0af6f7120dae2936f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:03.015288 26603 consensus_queue.cc:1035] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd154cd665be48f5b701775fed332f14" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:03.042791 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:03.036977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:32883 (local address 127.23.173.66:38291)
0927 22:15:03.037164 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:03.037172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:03.037633 (+ 461us) server_negotiation.cc:407] Connection header received
0927 22:15:03.037826 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:03.037832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:03.037904 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:03.038027 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:03.038437 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.039279 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.041100 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.041520 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.041662 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:03.042127 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:03.042219 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:03.042316 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:03.042428 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 22:15:03.042508 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 22:15:03.042883 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:03.036922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38291 (local address 127.23.173.65:32883)
0927 22:15:03.037522 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:03.037545 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:03.037697 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:03.038052 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:03.038057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:03.038070 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:03.038292 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:03.038300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.039403 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.039410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:03.040953 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:03.040973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:03.041700 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:03.041707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:03.041862 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:03.041905 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:03.042319 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:03.042333 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:15:03.042410 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:15:03.042543 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
W20260927 22:15:03.044116 26611 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:15:04.635685 25984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:04.635919 25984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:04.639940 25984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e18c89f2b614ceb931334342c0a844f: Sending DeleteTablet for 3 replicas of tablet 0219f2942dfe4e3b9d90690175572b68
I20260927 22:15:04.640632 26120 tablet_service.cc:1514] Processing DeleteTablet for tablet 0219f2942dfe4e3b9d90690175572b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:04 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33202
I20260927 22:15:04.640895 26401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0219f2942dfe4e3b9d90690175572b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:04 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:54156
I20260927 22:15:04.641331 26653 tablet_replica.cc:330] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: stopping tablet replica
I20260927 22:15:04.641394 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26055
I20260927 22:15:04.641381 26260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0219f2942dfe4e3b9d90690175572b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:04 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35810
I20260927 22:15:04.641530 26653 raft_consensus.cc:2229] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:04.641767 26653 raft_consensus.cc:2258] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:04.642575 26654 tablet_replica.cc:330] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: stopping tablet replica
I20260927 22:15:04.642853 26654 raft_consensus.cc:2229] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:04.642886 26653 ts_tablet_manager.cc:1905] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:04.643227 26654 raft_consensus.cc:2258] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:04.644925 26654 ts_tablet_manager.cc:1905] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:04.650655 26655 tablet_replica.cc:330] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: stopping tablet replica
I20260927 22:15:04.650943 26655 raft_consensus.cc:2229] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:04.651137 26653 ts_tablet_manager.cc:1918] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:04.651302 26653 log.cc:1196] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/0219f2942dfe4e3b9d90690175572b68
I20260927 22:15:04.651695 26655 raft_consensus.cc:2258] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:04.651921 26653 ts_tablet_manager.cc:1939] T 0219f2942dfe4e3b9d90690175572b68 P bd154cd665be48f5b701775fed332f14: Deleting consensus metadata
I20260927 22:15:04.652822 26655 ts_tablet_manager.cc:1905] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:04.653036 25968 catalog_manager.cc:4841] TS bd154cd665be48f5b701775fed332f14 (127.23.173.65:42895): tablet 0219f2942dfe4e3b9d90690175572b68 (table test-table [id=7efdbc84e12d4583be593a83ceaef047]) successfully deleted
I20260927 22:15:04.653842 26654 ts_tablet_manager.cc:1918] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:04.654006 26654 log.cc:1196] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/0219f2942dfe4e3b9d90690175572b68
I20260927 22:15:04.654426 26654 ts_tablet_manager.cc:1939] T 0219f2942dfe4e3b9d90690175572b68 P 71f9c41a406e41e0af6f7120dae2936f: Deleting consensus metadata
I20260927 22:15:04.655503 25968 catalog_manager.cc:4841] TS 71f9c41a406e41e0af6f7120dae2936f (127.23.173.67:37359): tablet 0219f2942dfe4e3b9d90690175572b68 (table test-table [id=7efdbc84e12d4583be593a83ceaef047]) successfully deleted
I20260927 22:15:04.658239 26655 ts_tablet_manager.cc:1918] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:15:04.658393 26655 log.cc:1196] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/0219f2942dfe4e3b9d90690175572b68
I20260927 22:15:04.658812 26655 ts_tablet_manager.cc:1939] T 0219f2942dfe4e3b9d90690175572b68 P a6aecc557fca4b7e9c283ed937977969: Deleting consensus metadata
I20260927 22:15:04.659651 25968 catalog_manager.cc:4841] TS a6aecc557fca4b7e9c283ed937977969 (127.23.173.66:38291): tablet 0219f2942dfe4e3b9d90690175572b68 (table test-table [id=7efdbc84e12d4583be593a83ceaef047]) successfully deleted
Sep 27 22:15:04 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:04.669612 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:04.643540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.23.173.65:42895)
0927 22:15:04.644097 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:04.644107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:04.650785 (+ 6678us) server_negotiation.cc:407] Connection header received
0927 22:15:04.650857 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:04.650863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:04.650947 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:04.651045 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:04.653163 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.654339 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.656039 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.656680 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.656813 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:04.664350 (+ 7537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:04.664364 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:04.664379 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:04.664416 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:04.667128 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.667783 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.667790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.667793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.667929 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.668272 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.668274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.668276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.668595 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:04.668821 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:04.669096 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:15:04.669248 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":797,"threads_started":1}
W20260927 22:15:04.670341 26186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33216
W20260927 22:15:04.671074 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:04.671271 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26055
I20260927 22:15:04.697376 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26195
Sep 27 22:15:04 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:04.713304 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:04.698215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.23.173.66:38291)
0927 22:15:04.698779 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:04.698785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:04.698806 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:15:04.698872 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:04.698879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:04.698958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:04.699070 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:04.700286 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.701614 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.702998 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.703418 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.703520 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:04.707571 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:04.707586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:04.707603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:04.707644 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:04.710865 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.711476 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.711484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.711486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.711619 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.711916 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.711920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.711922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.712278 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:04.712545 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:04.712699 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:15:04.712874 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":285,"threads_started":1}
W20260927 22:15:04.714108 26325 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35836
W20260927 22:15:04.714797 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26195: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:04.714926 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26195
I20260927 22:15:04.739705 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26334
Sep 27 22:15:04 dist-test-slave-3wc3 krb5kdc[25939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547301, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:04.755409 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:04.740433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.23.173.67:37359)
0927 22:15:04.741003 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:04.741007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:04.741022 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:15:04.741076 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:04.741080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:04.741137 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:04.741262 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:04.742666 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.743408 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.744682 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:04.745108 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:04.745243 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:04.749717 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:04.749733 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:04.749748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:04.749780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:04.752734 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.753386 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.753393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.753395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.753521 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:04.753841 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:04.753847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:04.753850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:04.754356 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:04.754653 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:04.754992 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:15:04.755155 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
W20260927 22:15:04.756104 26464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54186
W20260927 22:15:04.756690 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26334: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:04.756837 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26334
I20260927 22:15:04.779138 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 25952
I20260927 22:15:04.780339 26024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:04.910081 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 25952
2026-09-27T22:15:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8920 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:04 dist-test-slave-3wc3 krb5kdc[26663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:05 dist-test-slave-3wc3 krb5kdc[26663](info): set up 2 sockets
Sep 27 22:15:05 dist-test-slave-3wc3 krb5kdc[26663](info): commencing operation
krb5kdc: starting...
W20260927 22:15:05.990188 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:06 dist-test-slave-3wc3 krb5kdc[26663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:06.175777 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38497
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:06.491354 26676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:06.491616 26676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:06.491782 26676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:06.499773 26676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:06.499881 26676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:06.499966 26676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:06.500078 26676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:06.500173 26676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:06.509763 26676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:06.511080 26676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:06.512377 26676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:06.524468 26684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:06.524537 26682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:06.524514 26681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:06.525051 26676 server_base.cc:1034] running on GCE node
I20260927 22:15:06.525914 26676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:06.527303 26676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:06.528549 26676 hybrid_clock.cc:648] HybridClock initialized: now 1790547306528503 us; error 55 us; skew 500 ppm
I20260927 22:15:06.528898 26676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:06 dist-test-slave-3wc3 krb5kdc[26663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:06.531965 26676 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:06.533334 26676 webserver.cc:460] Webserver started at http://127.23.173.126:38045/ using document root <none> and password file <none>
I20260927 22:15:06.534080 26676 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:06.534236 26676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:06.534559 26676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:06.537438 26676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "a5b48042562345f4a33d0258216d0cde"
format_stamp: "Formatted at 2026-09-27 22:15:06 on dist-test-slave-3wc3"
server_key: "ab500efc2d9bb36f99eea822b4ee2d2a"
server_key_iv: "9d7fa0fb4bc03b805e59be7312eddbaa"
server_key_version: "encryptionkey@0"
I20260927 22:15:06.538256 26676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "a5b48042562345f4a33d0258216d0cde"
format_stamp: "Formatted at 2026-09-27 22:15:06 on dist-test-slave-3wc3"
server_key: "ab500efc2d9bb36f99eea822b4ee2d2a"
server_key_iv: "9d7fa0fb4bc03b805e59be7312eddbaa"
server_key_version: "encryptionkey@0"
I20260927 22:15:06.543388 26676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:15:06.546813 26691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:06.548166 26676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:06.548352 26676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "a5b48042562345f4a33d0258216d0cde"
format_stamp: "Formatted at 2026-09-27 22:15:06 on dist-test-slave-3wc3"
server_key: "ab500efc2d9bb36f99eea822b4ee2d2a"
server_key_iv: "9d7fa0fb4bc03b805e59be7312eddbaa"
server_key_version: "encryptionkey@0"
I20260927 22:15:06.548561 26676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:06.564147 26676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:06.568375 26676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:06.568674 26676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:06.586833 26676 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:38497
I20260927 22:15:06.586861 26742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:38497 every 8 connection(s)
I20260927 22:15:06.588156 26676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:06.591734 26743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:06.592649 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26676
I20260927 22:15:06.592864 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:06.593488 24245 external_mini_cluster.cc:1428] Setting key 817a24d607b19945b3c482089ec40700
I20260927 22:15:06.605630 26743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde: Bootstrap starting.
Sep 27 22:15:06 dist-test-slave-3wc3 krb5kdc[26663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:06.609184 26743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:06.610387 26743 log.cc:824] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:06.613116 26743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde: No bootstrap required, opened a new log
I20260927 22:15:06.616472 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:06.595483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.23.173.126:38497)
0927 22:15:06.596332 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:06.596343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:06.596392 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:15:06.598035 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:06.598055 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:06.598430 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:06.598739 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:06.600483 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:06.601784 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:06.603699 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:06.604344 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:06.604469 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:06.609048 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:06.609056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:06.609071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:06.609109 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:06.612438 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:06.613130 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:06.613136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:06.613142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:06.613352 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:06.613734 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:06.613740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:06.613744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:06.614314 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:06.614631 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:06.615096 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 22:15:06.615501 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":432,"threads_started":1}
I20260927 22:15:06.621502 26743 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } }
I20260927 22:15:06.621889 26743 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:06.622018 26743 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b48042562345f4a33d0258216d0cde, State: Initialized, Role: FOLLOWER
I20260927 22:15:06.622624 26743 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } }
I20260927 22:15:06.622937 26743 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:06.623085 26743 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:06.623214 26743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:06.625154 26743 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } }
I20260927 22:15:06.625566 26743 leader_election.cc:302] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b48042562345f4a33d0258216d0cde; no voters:
I20260927 22:15:06.626104 26743 leader_election.cc:288] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:06.626240 26748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:06.626907 26748 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [term 1 LEADER]: Becoming Leader. State: Replica: a5b48042562345f4a33d0258216d0cde, State: Running, Role: LEADER
I20260927 22:15:06.627409 26748 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } }
I20260927 22:15:06.628057 26743 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:06.630960 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b48042562345f4a33d0258216d0cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } } }
I20260927 22:15:06.631399 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:06.633747 26755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:06.633632 26750 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b48042562345f4a33d0258216d0cde. Latest consensus state: current_term: 1 leader_uuid: "a5b48042562345f4a33d0258216d0cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b48042562345f4a33d0258216d0cde" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38497 } } }
I20260927 22:15:06.633945 26750 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:06.638872 26755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:06.648234 26755 catalog_manager.cc:1347] Generated new cluster ID: 686c15b96cd84a09958b523b22dc6dba
I20260927 22:15:06.648347 26755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:06.674451 26755 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:06.676059 26755 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:06.685045 26755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b48042562345f4a33d0258216d0cde: Generated new TSK 0
I20260927 22:15:06.685906 26755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:06.745260 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:07.070528 26769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:07.070829 26769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:07.071003 26769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:07.078002 26769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:07.078142 26769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:07.078323 26769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:07.087564 26769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:07.088792 26769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:07.090128 26769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:07.102679 26777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:07.102649 26775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:07.102836 26774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:07.103564 26769 server_base.cc:1034] running on GCE node
I20260927 22:15:07.104203 26769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:07.105069 26769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:07.106287 26769 hybrid_clock.cc:648] HybridClock initialized: now 1790547307106253 us; error 46 us; skew 500 ppm
I20260927 22:15:07.106592 26769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:07 dist-test-slave-3wc3 krb5kdc[26663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547307, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:07.109711 26769 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:07.111093 26769 webserver.cc:460] Webserver started at http://127.23.173.65:46773/ using document root <none> and password file <none>
I20260927 22:15:07.111799 26769 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:07.111948 26769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:07.112210 26769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:07.114984 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "f678120fd29f4cc88a4ab31923a3eaca"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "d89f23fe3d1adc6107419a472f7cb362"
server_key_iv: "04bc09b00fe37415c2bcda9244670af6"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.115695 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "f678120fd29f4cc88a4ab31923a3eaca"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "d89f23fe3d1adc6107419a472f7cb362"
server_key_iv: "04bc09b00fe37415c2bcda9244670af6"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.120697 26769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:07.123965 26784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.125175 26769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:07.125339 26769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "f678120fd29f4cc88a4ab31923a3eaca"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "d89f23fe3d1adc6107419a472f7cb362"
server_key_iv: "04bc09b00fe37415c2bcda9244670af6"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.125535 26769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:07.138149 26769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:07.141999 26769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:07.142277 26769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:07.143146 26769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:07.144444 26769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:07.144567 26769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.144709 26769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:07.144783 26769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.177688 26769 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:41673
I20260927 22:15:07.177726 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:41673 every 8 connection(s)
I20260927 22:15:07.178987 26769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:07.185474 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26769
I20260927 22:15:07.185694 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:07.186367 24245 external_mini_cluster.cc:1428] Setting key f2b509d41730f64b2d6bb06d05569948
Sep 27 22:15:07 dist-test-slave-3wc3 krb5kdc[26663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547307, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:07.199636 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:07.181518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:33767 (local address 127.23.173.126:38497)
0927 22:15:07.182044 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:07.182055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:07.182901 (+ 846us) server_negotiation.cc:407] Connection header received
0927 22:15:07.184822 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:07.184828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:07.184914 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:07.185034 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:07.186736 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.187593 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.189194 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.189597 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.189730 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:07.193062 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:07.193067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:07.193070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:07.193100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:07.196225 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:07.197029 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:07.197040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:07.197044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:07.197166 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:07.197629 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:07.197638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:07.197641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:07.198074 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:07.198258 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:07.199173 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 22:15:07.199345 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":268,"threads_started":1}
I20260927 22:15:07.201082 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:07.182061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38497 (local address 127.23.173.65:33767)
0927 22:15:07.182734 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:07.182775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:07.184553 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:07.185273 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:07.185282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:07.185909 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:07.186575 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:07.186594 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.187737 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.187743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:07.189051 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:07.189063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.189813 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.189820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:07.190130 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:07.190790 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:07.190811 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:07.192842 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:07.196407 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:07.196417 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:07.196436 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:07.196886 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:07.197286 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:07.197293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:07.197297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:07.197497 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:07.198242 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:07.198255 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:07.198734 (+ 479us) client_negotiation.cc:769] Sending connection context
0927 22:15:07.199047 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 22:15:07.199613 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260927 22:15:07.202694 26897 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38497
I20260927 22:15:07.203051 26897 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:07.203742 26897 heartbeater.cc:511] Master 127.23.173.126:38497 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:07.205838 26708 ts_manager.cc:194] Registered new tserver with Master: f678120fd29f4cc88a4ab31923a3eaca (127.23.173.65:41673)
I20260927 22:15:07.207221 26708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:33767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:07.230198 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:07.542606 26903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:07.542927 26903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:07.543102 26903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:07.550098 26903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:07.550240 26903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:07.550431 26903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:07.559881 26903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:07.561107 26903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:07.562387 26903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:07.574862 26909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:07.575153 26908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:07.576903 26903 server_base.cc:1034] running on GCE node
W20260927 22:15:07.577016 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:07.577647 26903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:07.578846 26903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:07.580164 26903 hybrid_clock.cc:648] HybridClock initialized: now 1790547307580103 us; error 43 us; skew 500 ppm
I20260927 22:15:07.580514 26903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:07 dist-test-slave-3wc3 krb5kdc[26663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547307, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:07.583694 26903 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:07.585196 26903 webserver.cc:460] Webserver started at http://127.23.173.66:42955/ using document root <none> and password file <none>
I20260927 22:15:07.585865 26903 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:07.586036 26903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:07.586385 26903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:07.589114 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "f74df33fbba1408fad6a0c54904d471e"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "791f49868a99d8808a5ab5d1b7dc1553"
server_key_iv: "59f454601a0fca51e7f7cd3904f5e247"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.589879 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "f74df33fbba1408fad6a0c54904d471e"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "791f49868a99d8808a5ab5d1b7dc1553"
server_key_iv: "59f454601a0fca51e7f7cd3904f5e247"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.594959 26903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:15:07.598457 26918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.599846 26903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:15:07.600031 26903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "f74df33fbba1408fad6a0c54904d471e"
format_stamp: "Formatted at 2026-09-27 22:15:07 on dist-test-slave-3wc3"
server_key: "791f49868a99d8808a5ab5d1b7dc1553"
server_key_iv: "59f454601a0fca51e7f7cd3904f5e247"
server_key_version: "encryptionkey@0"
I20260927 22:15:07.600224 26903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:07.617172 26903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:07.621874 26903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:07.622213 26903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:07.623205 26903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:07.624536 26903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:07.624645 26903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.624816 26903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:07.624895 26903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:07.659548 26903 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:45767
I20260927 22:15:07.659590 27030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:45767 every 8 connection(s)
I20260927 22:15:07.660854 26903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:07.666742 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 26903
I20260927 22:15:07.666934 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:07.667474 24245 external_mini_cluster.cc:1428] Setting key 533563aca0b3f2aaa0709ffb9df63f79
Sep 27 22:15:07 dist-test-slave-3wc3 krb5kdc[26663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547307, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:07.682520 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:07.663322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38621 (local address 127.23.173.126:38497)
0927 22:15:07.663499 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:07.663503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:07.664615 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 22:15:07.666361 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:07.666365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:07.666436 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:07.666523 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:07.668490 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.669610 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.672108 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.672511 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.672613 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:07.676663 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:07.676669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:07.676672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:07.676702 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:07.679279 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:07.680198 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:07.680201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:07.680203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:07.680316 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:07.680804 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:07.680809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:07.680812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:07.681092 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:07.681206 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:07.682101 (+ 895us) server_negotiation.cc:299] Negotiation successful
0927 22:15:07.682265 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:15:07.684036 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:07.663755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38497 (local address 127.23.173.66:38621)
0927 22:15:07.664428 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:07.664465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:07.666103 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:07.666801 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:07.666813 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:07.667554 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:07.668289 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:07.668314 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.669764 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.669769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:07.671932 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:07.671950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:07.672739 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:07.672748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:07.673065 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:07.673781 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:07.673818 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:07.676465 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:07.679421 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:07.679432 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:07.679451 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:07.680084 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:07.680425 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:07.680432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:07.680438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:07.680676 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:07.681218 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:07.681228 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:07.681677 (+ 449us) client_negotiation.cc:769] Sending connection context
0927 22:15:07.682000 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 22:15:07.682570 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":258,"threads_started":1}
I20260927 22:15:07.685508 27031 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38497
I20260927 22:15:07.685832 27031 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:07.686604 27031 heartbeater.cc:511] Master 127.23.173.126:38497 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:15:07.688002 26708 ts_manager.cc:194] Registered new tserver with Master: f74df33fbba1408fad6a0c54904d471e (127.23.173.66:45767)
I20260927 22:15:07.688885 26708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:38621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:07.713243 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:08.029132 27036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:08.029438 27036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:08.029603 27036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:08.036970 27036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:08.037113 27036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:08.037294 27036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:08.046754 27036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:08.047976 27036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:08.049286 27036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:08.064527 27041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:08.064567 27042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:08.064599 27044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:08.065423 27036 server_base.cc:1034] running on GCE node
I20260927 22:15:08.066071 27036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:08.066941 27036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:08.068164 27036 hybrid_clock.cc:648] HybridClock initialized: now 1790547308068125 us; error 48 us; skew 500 ppm
I20260927 22:15:08.068465 27036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:08 dist-test-slave-3wc3 krb5kdc[26663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547308, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:08.071605 27036 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:08.073082 27036 webserver.cc:460] Webserver started at http://127.23.173.67:44921/ using document root <none> and password file <none>
I20260927 22:15:08.073796 27036 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:08.073904 27036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:08.074221 27036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:08.076963 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "1723e105ff474f0da5de347977783a8d"
format_stamp: "Formatted at 2026-09-27 22:15:08 on dist-test-slave-3wc3"
server_key: "a46d1ea749ae855eff7085a76b2a9728"
server_key_iv: "88f587c0987c28c48095f4be1dface84"
server_key_version: "encryptionkey@0"
I20260927 22:15:08.077658 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "1723e105ff474f0da5de347977783a8d"
format_stamp: "Formatted at 2026-09-27 22:15:08 on dist-test-slave-3wc3"
server_key: "a46d1ea749ae855eff7085a76b2a9728"
server_key_iv: "88f587c0987c28c48095f4be1dface84"
server_key_version: "encryptionkey@0"
I20260927 22:15:08.082540 27036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:15:08.085807 27051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:08.087132 27036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:08.087252 27036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "1723e105ff474f0da5de347977783a8d"
format_stamp: "Formatted at 2026-09-27 22:15:08 on dist-test-slave-3wc3"
server_key: "a46d1ea749ae855eff7085a76b2a9728"
server_key_iv: "88f587c0987c28c48095f4be1dface84"
server_key_version: "encryptionkey@0"
I20260927 22:15:08.087409 27036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:08.130194 27036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:08.134123 27036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:08.134366 27036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:08.135339 27036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:08.136660 27036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:08.136731 27036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:08.136818 27036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:08.136862 27036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:08.169940 27036 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43003
I20260927 22:15:08.170032 27163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43003 every 8 connection(s)
I20260927 22:15:08.171244 27036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:08.181253 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27036
I20260927 22:15:08.181459 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:08.181896 24245 external_mini_cluster.cc:1428] Setting key 8e47348d6384af74d55aaf8d4100bd02
Sep 27 22:15:08 dist-test-slave-3wc3 krb5kdc[26663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547308, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:08.191201 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:08.174028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34707 (local address 127.23.173.126:38497)
0927 22:15:08.174272 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:08.174278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:08.175376 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 22:15:08.177205 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:08.177211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:08.177332 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:08.177448 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:08.179010 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:08.179762 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:08.181568 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:08.182085 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:08.182186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:08.185705 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:08.185709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:08.185712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:08.185741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:08.188272 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:08.189040 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:08.189043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:08.189046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:08.189162 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:08.189559 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:08.189562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:08.189564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:08.189813 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:08.190032 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:08.190676 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:15:08.190918 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:15:08.191871 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:08.174490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38497 (local address 127.23.173.67:34707)
0927 22:15:08.175210 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:08.175254 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:08.176920 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:08.177699 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:08.177708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:08.178293 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:08.178841 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:08.178859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:08.179920 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:08.179930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:08.181388 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:08.181406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:08.182316 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:08.182324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:08.182587 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:08.183308 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:08.183334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:08.185537 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:08.188395 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:08.188401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:08.188411 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:08.188940 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:08.189284 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:08.189288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:08.189291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:08.189461 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:08.190051 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:08.190056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:08.190334 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 22:15:08.190550 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 22:15:08.190888 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":304,"threads_started":1}
I20260927 22:15:08.193104 27164 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38497
I20260927 22:15:08.193380 27164 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:08.194092 27164 heartbeater.cc:511] Master 127.23.173.126:38497 requested a full tablet report, sending...
I20260927 22:15:08.195403 26708 ts_manager.cc:194] Registered new tserver with Master: 1723e105ff474f0da5de347977783a8d (127.23.173.67:43003)
I20260927 22:15:08.196537 26708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:34707
I20260927 22:15:08.197129 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:08.210090 26897 heartbeater.cc:503] Master 127.23.173.126:38497 was elected leader, sending a full tablet report...
W20260927 22:15:08.226141 26899 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:08.223670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.23.173.126:38497)
0927 22:15:08.223960 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:08.223967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:08.224226 (+ 259us) server_negotiation.cc:407] Connection header received
0927 22:15:08.224436 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:08.224446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:08.224507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:08.224612 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:08.225807 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48136: BlockingRecv error: recv got EOF from 127.0.0.1:48136 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:15:08.229151 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26769
W20260927 22:15:08.231565 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26769: Not authorized: Client connection negotiation failed: client connection to 127.23.173.65:41673: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:15:08.231751 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26769
I20260927 22:15:08.251853 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26903
W20260927 22:15:08.254763 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26903: Not authorized: Client connection negotiation failed: client connection to 127.23.173.66:45767: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:15:08.254916 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26903
I20260927 22:15:08.276548 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27036
W20260927 22:15:08.279973 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27036: Not authorized: Client connection negotiation failed: client connection to 127.23.173.67:43003: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:15:08.280115 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27036
I20260927 22:15:08.296343 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 26676
I20260927 22:15:08.297524 26738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:08.399947 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 26676
2026-09-27T22:15:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3486 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:08 dist-test-slave-3wc3 krb5kdc[27181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:08 dist-test-slave-3wc3 krb5kdc[27181](info): set up 2 sockets
Sep 27 22:15:08 dist-test-slave-3wc3 krb5kdc[27181](info): commencing operation
krb5kdc: starting...
W20260927 22:15:09.474678 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:09 dist-test-slave-3wc3 krb5kdc[27181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:09.667246 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:45687
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:09.983645 27194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:09.983929 27194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:09.984105 27194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:09.991314 27194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:09.991421 27194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:09.991529 27194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:09.991621 27194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:09.991696 27194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:10.001492 27194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:45687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:10.002887 27194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:10.004206 27194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:10.016100 27202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:10.016187 27200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:10.016214 27199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:10.017326 27194 server_base.cc:1034] running on GCE node
I20260927 22:15:10.018136 27194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:10.019503 27194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:10.020730 27194 hybrid_clock.cc:648] HybridClock initialized: now 1790547310020694 us; error 62 us; skew 500 ppm
I20260927 22:15:10.021067 27194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:10 dist-test-slave-3wc3 krb5kdc[27181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547310, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:10.031401 27194 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:10.032778 27194 webserver.cc:460] Webserver started at http://127.23.173.126:42707/ using document root <none> and password file <none>
I20260927 22:15:10.033392 27194 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:10.033497 27194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:10.033747 27194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:10.036641 27194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "9844aff251244902a1c0229aefa78a2e"
format_stamp: "Formatted at 2026-09-27 22:15:10 on dist-test-slave-3wc3"
server_key: "644b4b0265652b532609f1784b49bb17"
server_key_iv: "2f6a881e0a1e199e010b7ee02d6d426c"
server_key_version: "encryptionkey@0"
I20260927 22:15:10.037333 27194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "9844aff251244902a1c0229aefa78a2e"
format_stamp: "Formatted at 2026-09-27 22:15:10 on dist-test-slave-3wc3"
server_key: "644b4b0265652b532609f1784b49bb17"
server_key_iv: "2f6a881e0a1e199e010b7ee02d6d426c"
server_key_version: "encryptionkey@0"
I20260927 22:15:10.042253 27194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:15:10.045446 27209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:10.046741 27194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:10.046864 27194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9844aff251244902a1c0229aefa78a2e"
format_stamp: "Formatted at 2026-09-27 22:15:10 on dist-test-slave-3wc3"
server_key: "644b4b0265652b532609f1784b49bb17"
server_key_iv: "2f6a881e0a1e199e010b7ee02d6d426c"
server_key_version: "encryptionkey@0"
I20260927 22:15:10.047019 27194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:10.071974 27194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:10.076397 27194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:10.076651 27194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:10.096556 27194 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:45687
I20260927 22:15:10.096586 27260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:45687 every 8 connection(s)
I20260927 22:15:10.098008 27194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:10.101708 27261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:10.105010 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27194
I20260927 22:15:10.105239 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:10.106057 24245 external_mini_cluster.cc:1428] Setting key 4e6161284f4f01790c23db526163913d
I20260927 22:15:10.114739 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e: Bootstrap starting.
I20260927 22:15:10.118659 27261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:10.120072 27261 log.cc:824] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:15:10 dist-test-slave-3wc3 krb5kdc[27181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:10.123090 27261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e: No bootstrap required, opened a new log
I20260927 22:15:10.129156 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:10.107926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.23.173.126:45687)
0927 22:15:10.108710 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:10.108721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:10.108761 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:15:10.110471 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:10.110508 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:10.110910 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:10.111240 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:10.113110 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:10.114452 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:10.116180 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:10.116886 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:10.117025 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:10.121909 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:10.121916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:10.121930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:10.122020 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:10.125037 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:10.125880 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:10.125887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:10.125893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:10.126129 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:10.126558 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:10.126561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:10.126563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:10.127075 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:10.127289 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:10.127763 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 22:15:10.128128 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":373,"threads_started":1}
I20260927 22:15:10.132467 27261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } }
I20260927 22:15:10.132848 27261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:10.132978 27261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9844aff251244902a1c0229aefa78a2e, State: Initialized, Role: FOLLOWER
I20260927 22:15:10.133060 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:15:10.133592 27261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } }
I20260927 22:15:10.133834 27261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:10.134049 27261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:10.134219 27261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:10.136308 27261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } }
I20260927 22:15:10.136790 27261 leader_election.cc:302] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9844aff251244902a1c0229aefa78a2e; no voters:
I20260927 22:15:10.137367 27261 leader_election.cc:288] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:10.137507 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:10.138201 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [term 1 LEADER]: Becoming Leader. State: Replica: 9844aff251244902a1c0229aefa78a2e, State: Running, Role: LEADER
I20260927 22:15:10.138808 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } }
I20260927 22:15:10.140866 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9844aff251244902a1c0229aefa78a2e. Latest consensus state: current_term: 1 leader_uuid: "9844aff251244902a1c0229aefa78a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } } }
I20260927 22:15:10.141294 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:10.141502 27261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:10.141716 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9844aff251244902a1c0229aefa78a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9844aff251244902a1c0229aefa78a2e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 45687 } } }
I20260927 22:15:10.142086 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:10.147163 27271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:10.152673 27271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:10.166687 27271 catalog_manager.cc:1347] Generated new cluster ID: d46adcecb1d544608d8167560de1a20d
I20260927 22:15:10.166910 27271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:10.178017 27271 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:10.180392 27271 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:10.194896 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:10.184085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.23.173.126:45687)
0927 22:15:10.184240 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:10.184247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:10.186553 (+ 2306us) server_negotiation.cc:407] Connection header received
0927 22:15:10.186714 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:10.186720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:10.186802 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:10.186918 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:10.188089 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:10.189246 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:10.191130 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:10.191823 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:10.191971 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:10.193138 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:10.193146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:10.193151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:10.194038 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:10.194120 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:10.194162 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:10.194263 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:10.194424 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:15:10.194538 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:15:10.199069 27271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9844aff251244902a1c0229aefa78a2e: Generated new TSK 0
I20260927 22:15:10.199854 27271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:15:10.209899 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27194
I20260927 22:15:10.211202 27256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:10.299876 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27194
2026-09-27T22:15:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1892 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:10 dist-test-slave-3wc3 krb5kdc[27295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:10 dist-test-slave-3wc3 krb5kdc[27295](info): set up 2 sockets
Sep 27 22:15:10 dist-test-slave-3wc3 krb5kdc[27295](info): commencing operation
krb5kdc: starting...
W20260927 22:15:11.406344 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:11 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:11.599593 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:36837
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:11.918936 27308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:11.919225 27308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:11.919382 27308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:11.926666 27308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:11.926767 27308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:11.926885 27308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:11.926970 27308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:11.927043 27308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:11.937057 27308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:11.938462 27308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:11.939780 27308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:11.954190 27314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:11.954232 27316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:11.954269 27313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:11.955152 27308 server_base.cc:1034] running on GCE node
I20260927 22:15:11.955989 27308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:11.957347 27308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:11.958583 27308 hybrid_clock.cc:648] HybridClock initialized: now 1790547311958536 us; error 60 us; skew 500 ppm
I20260927 22:15:11.958915 27308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:11 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:11.961898 27308 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:11.963279 27308 webserver.cc:460] Webserver started at http://127.23.173.126:39079/ using document root <none> and password file <none>
I20260927 22:15:11.963881 27308 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:11.963974 27308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:11.964206 27308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:11.967175 27308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "e9ae15ac8b0949c5881052292d399e38"
format_stamp: "Formatted at 2026-09-27 22:15:11 on dist-test-slave-3wc3"
server_key: "5c2e3cd76d51c2bdfed56cdec38ff534"
server_key_iv: "76d468cd9b114ad6ca166606007fecba"
server_key_version: "encryptionkey@0"
I20260927 22:15:11.967867 27308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "e9ae15ac8b0949c5881052292d399e38"
format_stamp: "Formatted at 2026-09-27 22:15:11 on dist-test-slave-3wc3"
server_key: "5c2e3cd76d51c2bdfed56cdec38ff534"
server_key_iv: "76d468cd9b114ad6ca166606007fecba"
server_key_version: "encryptionkey@0"
I20260927 22:15:11.972913 27308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:15:11.976250 27323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:11.977577 27308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:11.977743 27308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "e9ae15ac8b0949c5881052292d399e38"
format_stamp: "Formatted at 2026-09-27 22:15:11 on dist-test-slave-3wc3"
server_key: "5c2e3cd76d51c2bdfed56cdec38ff534"
server_key_iv: "76d468cd9b114ad6ca166606007fecba"
server_key_version: "encryptionkey@0"
I20260927 22:15:11.977916 27308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:11.993081 27308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:11.997155 27308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:11.997439 27308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:12.016433 27308 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:36837
I20260927 22:15:12.016491 27374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:36837 every 8 connection(s)
I20260927 22:15:12.017874 27308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:12.022172 27375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:12.027163 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27308
I20260927 22:15:12.027405 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:12.028013 24245 external_mini_cluster.cc:1428] Setting key 760416fd477be897d4ff46f4e9a5df1e
I20260927 22:15:12.034090 27375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38: Bootstrap starting.
I20260927 22:15:12.037045 27375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:12.038120 27375 log.cc:824] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:12.040812 27375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38: No bootstrap required, opened a new log
Sep 27 22:15:12 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:12.048928 27375 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } }
I20260927 22:15:12.049322 27375 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:12.049451 27375 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ae15ac8b0949c5881052292d399e38, State: Initialized, Role: FOLLOWER
I20260927 22:15:12.049692 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:12.030002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.23.173.126:36837)
0927 22:15:12.030855 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:12.030866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:12.030907 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:15:12.032695 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:12.032729 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:12.033107 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:12.033465 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:12.034875 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.036146 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.037420 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.038129 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.038315 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:12.042338 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:12.042346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:12.042359 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:12.042400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:12.045532 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:12.046267 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:12.046281 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:12.046293 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:12.046458 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:12.046852 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:12.046857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:12.046859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:12.047384 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:12.047725 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:12.048186 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 22:15:12.048566 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":399,"threads_started":1}
I20260927 22:15:12.050144 27375 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } }
I20260927 22:15:12.050441 27375 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:12.050585 27375 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:12.050758 27375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:12.054020 27375 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } }
I20260927 22:15:12.054510 27375 leader_election.cc:302] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9ae15ac8b0949c5881052292d399e38; no voters:
I20260927 22:15:12.055169 27375 leader_election.cc:288] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:12.055269 27380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:12.056625 27380 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [term 1 LEADER]: Becoming Leader. State: Replica: e9ae15ac8b0949c5881052292d399e38, State: Running, Role: LEADER
I20260927 22:15:12.057233 27380 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } }
I20260927 22:15:12.058487 27375 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:12.060029 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9ae15ac8b0949c5881052292d399e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } } }
I20260927 22:15:12.060374 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:12.061687 27382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ae15ac8b0949c5881052292d399e38. Latest consensus state: current_term: 1 leader_uuid: "e9ae15ac8b0949c5881052292d399e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae15ac8b0949c5881052292d399e38" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36837 } } }
I20260927 22:15:12.062036 27382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:12.067521 27385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:12.072863 27385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:12.082178 27385 catalog_manager.cc:1347] Generated new cluster ID: 54b009eaea4f4d83b49088674448a829
I20260927 22:15:12.082299 27385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:12.101744 27385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:12.103056 27385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:12.108248 27385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9ae15ac8b0949c5881052292d399e38: Generated new TSK 0
I20260927 22:15:12.109055 27385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:12.169241 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:12.490867 27401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:12.491128 27401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:12.491302 27401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:12.498477 27401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:12.498576 27401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:12.498776 27401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:12.508567 27401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:12.509820 27401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:12.511119 27401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:12.523598 27406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:12.523696 27409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:12.523691 27407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:12.524732 27401 server_base.cc:1034] running on GCE node
I20260927 22:15:12.525305 27401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:12.526281 27401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:12.527503 27401 hybrid_clock.cc:648] HybridClock initialized: now 1790547312527462 us; error 36 us; skew 500 ppm
I20260927 22:15:12.527810 27401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:12 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:12.531080 27401 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:12.532492 27401 webserver.cc:460] Webserver started at http://127.23.173.65:38697/ using document root <none> and password file <none>
I20260927 22:15:12.533113 27401 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:12.533210 27401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:12.533453 27401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:12.536281 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "eb7f696c58be46669fcc7f1bb2205b61"
format_stamp: "Formatted at 2026-09-27 22:15:12 on dist-test-slave-3wc3"
server_key: "35b3f254bc9fe5607dc1edfc09e9fd71"
server_key_iv: "7c656e6761fb13b88bf0d440159d8af8"
server_key_version: "encryptionkey@0"
I20260927 22:15:12.536983 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7f696c58be46669fcc7f1bb2205b61"
format_stamp: "Formatted at 2026-09-27 22:15:12 on dist-test-slave-3wc3"
server_key: "35b3f254bc9fe5607dc1edfc09e9fd71"
server_key_iv: "7c656e6761fb13b88bf0d440159d8af8"
server_key_version: "encryptionkey@0"
I20260927 22:15:12.541925 27401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:15:12.545213 27416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:12.546680 27401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:12.546811 27401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "eb7f696c58be46669fcc7f1bb2205b61"
format_stamp: "Formatted at 2026-09-27 22:15:12 on dist-test-slave-3wc3"
server_key: "35b3f254bc9fe5607dc1edfc09e9fd71"
server_key_iv: "7c656e6761fb13b88bf0d440159d8af8"
server_key_version: "encryptionkey@0"
I20260927 22:15:12.546972 27401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:12.558663 27401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:12.563009 27401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:12.563266 27401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:12.564147 27401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:12.565420 27401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:12.565493 27401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:12.565582 27401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:12.565626 27401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:12.600029 27401 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:38323
I20260927 22:15:12.600107 27528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:38323 every 8 connection(s)
I20260927 22:15:12.601326 27401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:12.607489 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27401
I20260927 22:15:12.607707 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:12.608413 24245 external_mini_cluster.cc:1428] Setting key 1f99d87e96b5cf4a57ebc7d623c3d75b
Sep 27 22:15:12 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:12.622609 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:12.604018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:53775 (local address 127.23.173.126:36837)
0927 22:15:12.604477 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:12.604482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:12.605274 (+ 792us) server_negotiation.cc:407] Connection header received
0927 22:15:12.607601 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:12.607607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:12.607702 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:12.607826 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:12.609703 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.610711 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.612281 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.612680 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.612811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:12.616848 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:12.616853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:12.616856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:12.616886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:12.619406 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:12.620321 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:12.620324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:12.620326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:12.620440 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:12.620907 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:12.620911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:12.620913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:12.621156 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:12.621273 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:12.622177 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 22:15:12.622336 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":250,"threads_started":1}
I20260927 22:15:12.624207 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:12.604429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:36837 (local address 127.23.173.65:53775)
0927 22:15:12.605118 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:12.605160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:12.607299 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:12.608239 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:12.608248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:12.608824 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:12.609527 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:12.609547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.610891 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.610895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:12.612123 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:12.612139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:12.612924 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:12.612933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:12.613146 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:12.613813 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:12.613844 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:12.616619 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:12.619543 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:12.619555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:12.619574 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:12.620202 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:12.620542 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:12.620550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:12.620556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:12.620798 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:12.621287 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:12.621298 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:12.621740 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 22:15:12.622091 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 22:15:12.622686 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":269,"threads_started":1}
I20260927 22:15:12.625801 27529 heartbeater.cc:347] Connected to a master server at 127.23.173.126:36837
I20260927 22:15:12.626178 27529 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:12.627102 27529 heartbeater.cc:511] Master 127.23.173.126:36837 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:12.629348 27340 ts_manager.cc:194] Registered new tserver with Master: eb7f696c58be46669fcc7f1bb2205b61 (127.23.173.65:38323)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:15:12.631327 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:53775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:12.653590 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:12.968609 27535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:12.968918 27535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:12.969098 27535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:12.976228 27535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:12.976333 27535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:12.976505 27535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:12.985828 27535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:12.987035 27535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:12.988304 27535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:13.000465 27540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:13.000591 27541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:13.000674 27543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:13.001534 27535 server_base.cc:1034] running on GCE node
I20260927 22:15:13.002118 27535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:13.002936 27535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:13.004129 27535 hybrid_clock.cc:648] HybridClock initialized: now 1790547313004084 us; error 45 us; skew 500 ppm
I20260927 22:15:13.004428 27535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:13.007543 27535 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:13.008957 27535 webserver.cc:460] Webserver started at http://127.23.173.66:45307/ using document root <none> and password file <none>
I20260927 22:15:13.009663 27535 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:13.009765 27535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:13.010102 27535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:13.013208 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "9b28b79cb9f24b81ba6ff705ccf202df"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "5d8aeae382f48114561acda2020bc856"
server_key_iv: "f02eb9f89d1108b4167d2501dda95d6e"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.013907 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "9b28b79cb9f24b81ba6ff705ccf202df"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "5d8aeae382f48114561acda2020bc856"
server_key_iv: "f02eb9f89d1108b4167d2501dda95d6e"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.019091 27535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:15:13.022780 27550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:13.024303 27535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:13.024473 27535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "9b28b79cb9f24b81ba6ff705ccf202df"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "5d8aeae382f48114561acda2020bc856"
server_key_iv: "f02eb9f89d1108b4167d2501dda95d6e"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.024674 27535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:13.035552 27535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:13.039335 27535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:13.039623 27535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:13.040493 27535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:13.041749 27535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:13.041865 27535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:13.042030 27535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:13.042107 27535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:15:13.075114 27535 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:37451
I20260927 22:15:13.075158 27662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:37451 every 8 connection(s)
I20260927 22:15:13.076380 27535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:13.081038 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27535
I20260927 22:15:13.081229 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:13.081758 24245 external_mini_cluster.cc:1428] Setting key 77a0c0c9a8deab3e7c30e7882821e27c
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:13.097525 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.078963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:48907 (local address 127.23.173.126:36837)
0927 22:15:13.079172 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.079176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.080345 (+ 1169us) server_negotiation.cc:407] Connection header received
0927 22:15:13.082420 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.082426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.082543 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.082646 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.084462 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.085706 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.087277 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.087987 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.088126 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.091938 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.091943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.091946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.091978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.094551 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.095267 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.095271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.095273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.095426 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.095861 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.095865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.095867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.096111 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.096267 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.096944 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.097168 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:15:13.098567 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.079459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:36837 (local address 127.23.173.66:48907)
0927 22:15:13.080175 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:13.080218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:13.082125 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:13.082928 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:13.082938 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:13.083679 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:13.084278 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:13.084296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.085857 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.085860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:13.087129 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:13.087140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.088179 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.088183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:13.088365 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.089060 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:13.089099 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:13.091715 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:13.094692 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:13.094701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:13.094717 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:13.095132 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:13.095542 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:13.095549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:13.095553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:13.095736 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:13.096248 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:13.096256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:13.096581 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:15:13.096838 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:15:13.097358 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":280,"threads_started":1}
I20260927 22:15:13.100131 27663 heartbeater.cc:347] Connected to a master server at 127.23.173.126:36837
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:15:13.100436 27663 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:13.101125 27663 heartbeater.cc:511] Master 127.23.173.126:36837 requested a full tablet report, sending...
I20260927 22:15:13.102626 27340 ts_manager.cc:194] Registered new tserver with Master: 9b28b79cb9f24b81ba6ff705ccf202df (127.23.173.66:37451)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
I20260927 22:15:13.103608 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:48907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:13.125921 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:13.452996 27668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:13.453277 27668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:13.453439 27668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:13.460659 27668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:13.460790 27668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:13.461006 27668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:13.470700 27668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:13.471994 27668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:13.473287 27668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:13.485992 27673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:13.486002 27674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:13.486123 27676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:13.486817 27668 server_base.cc:1034] running on GCE node
I20260927 22:15:13.487327 27668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:13.488184 27668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:13.489373 27668 hybrid_clock.cc:648] HybridClock initialized: now 1790547313489338 us; error 30 us; skew 500 ppm
I20260927 22:15:13.489702 27668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:13.492877 27668 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:13.494328 27668 webserver.cc:460] Webserver started at http://127.23.173.67:33463/ using document root <none> and password file <none>
I20260927 22:15:13.494974 27668 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:13.495116 27668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:13.495502 27668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:13.498382 27668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "6091f1701b7d4bcfb05fcde7ba1e7a42"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "c6ec4a0fc8828b0b80ee03672e80e1c9"
server_key_iv: "151792839eb19f6f86043132335f344c"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.499154 27668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "6091f1701b7d4bcfb05fcde7ba1e7a42"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "c6ec4a0fc8828b0b80ee03672e80e1c9"
server_key_iv: "151792839eb19f6f86043132335f344c"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.504117 27668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:15:13.507380 27683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:13.508574 27668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:15:13.508742 27668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "6091f1701b7d4bcfb05fcde7ba1e7a42"
format_stamp: "Formatted at 2026-09-27 22:15:13 on dist-test-slave-3wc3"
server_key: "c6ec4a0fc8828b0b80ee03672e80e1c9"
server_key_iv: "151792839eb19f6f86043132335f344c"
server_key_version: "encryptionkey@0"
I20260927 22:15:13.508932 27668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:13.534924 27668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:13.539160 27668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:13.539472 27668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:13.540377 27668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:13.541662 27668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:13.541766 27668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:13.541925 27668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:13.542023 27668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:15:13.574857 27668 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:33555
I20260927 22:15:13.574899 27795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:33555 every 8 connection(s)
I20260927 22:15:13.576143 27668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:13.584095 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27668
I20260927 22:15:13.584273 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:13.584892 24245 external_mini_cluster.cc:1428] Setting key ecc66025e2a8a121aac4294d04aacbe3
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:13.595784 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.578732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:56193 (local address 127.23.173.126:36837)
0927 22:15:13.578941 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.578948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.580125 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 22:15:13.581908 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.581912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.582058 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.582157 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.583531 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.584333 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.586527 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.586963 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.587102 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.590486 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.590491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.590494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.590524 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.592828 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.593537 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.593549 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.593553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.593665 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.594134 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.594142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.594146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.594410 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.594565 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.595228 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.595438 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:15:13.596681 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.579283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:36837 (local address 127.23.173.67:56193)
0927 22:15:13.579955 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:13.579991 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:13.581600 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:13.582347 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:13.582354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:13.582831 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:13.583358 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:13.583373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.584505 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.584511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:13.586381 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:13.586401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.587197 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.587205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:13.587456 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.588130 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:13.588151 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:13.590265 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:13.592977 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:13.592985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:13.592999 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:13.593408 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:13.593776 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:13.593781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:13.593784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:13.594015 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:13.594537 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:13.594544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:13.594872 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:15:13.595121 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 22:15:13.595607 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":268,"threads_started":1}
I20260927 22:15:13.597867 27796 heartbeater.cc:347] Connected to a master server at 127.23.173.126:36837
I20260927 22:15:13.598219 27796 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:13.599176 27796 heartbeater.cc:511] Master 127.23.173.126:36837 requested a full tablet report, sending...
I20260927 22:15:13.600735 27340 ts_manager.cc:194] Registered new tserver with Master: 6091f1701b7d4bcfb05fcde7ba1e7a42 (127.23.173.67:33555)
I20260927 22:15:13.601822 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:56193
I20260927 22:15:13.612092 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:13.634877 27529 heartbeater.cc:503] Master 127.23.173.126:36837 was elected leader, sending a full tablet report...
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:13.659412 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.645632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.23.173.126:36837)
0927 22:15:13.645868 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.645875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.646316 (+ 441us) server_negotiation.cc:407] Connection header received
0927 22:15:13.646537 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.646542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.646603 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.646712 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.648035 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.648765 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.650144 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.650577 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.650731 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.654464 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.654469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.654472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.654502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.656821 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.657451 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.657463 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.657466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.657593 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.657992 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.657997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.658001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.658337 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.658595 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.658854 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.659076 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:15:13.660059 27340 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43126
I20260927 22:15:13.662041 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27401
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:13.677201 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.663050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.23.173.65:38323)
0927 22:15:13.663696 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.663703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.663727 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:13.663780 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.663786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.664014 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.664130 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.665247 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.666329 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.667556 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.668206 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.668368 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.672137 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.672147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.672160 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.672192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.674830 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.675426 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.675432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.675434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.675551 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.675838 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.675841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.675843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.676176 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.676384 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.676619 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.676797 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":334,"threads_started":1}
W20260927 22:15:13.678503 27524 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57196
W20260927 22:15:13.679556 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27401: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:13.679768 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27401
I20260927 22:15:13.695792 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27535
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:13.711455 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.696790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.23.173.66:37451)
0927 22:15:13.697297 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.697304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.697328 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:13.697391 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.697397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.697604 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.697729 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.698931 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.699943 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.701133 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.701687 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.701812 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.706005 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.706034 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.706060 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.706095 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.708842 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.709428 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.709434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.709437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.709548 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.709856 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.709859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.709861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.710314 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.710545 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.710843 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.711133 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
W20260927 22:15:13.712767 27658 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37772
W20260927 22:15:13.713872 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:13.714080 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27535
I20260927 22:15:13.729434 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27668
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547313, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:13.745705 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:13.730393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.23.173.67:33555)
0927 22:15:13.731034 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:13.731041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:13.731071 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:15:13.731131 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:13.731137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:13.731350 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:13.731483 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:13.732878 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.734149 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.735373 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:13.735985 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:13.736091 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:13.740341 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:13.740352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:13.740366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:13.740409 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:13.743323 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.743938 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.743945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.743947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.744072 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:13.744364 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:13.744367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:13.744369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:13.744722 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:13.745000 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:13.745216 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:15:13.745418 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":326,"threads_started":1}
W20260927 22:15:13.747018 27791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40784
W20260927 22:15:13.748076 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:15:13.748250 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27668
I20260927 22:15:13.763947 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27308
I20260927 22:15:13.765098 27370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:13.870035 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27308
2026-09-27T22:15:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3578 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27813](info): set up 2 sockets
Sep 27 22:15:13 dist-test-slave-3wc3 krb5kdc[27813](info): commencing operation
krb5kdc: starting...
W20260927 22:15:14.958441 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:15 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:15.149840 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:33295
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:15.458582 27826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:15.458925 27826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:15.459105 27826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:15.466267 27826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:15.466418 27826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:15.466532 27826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:15.466641 27826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:15.466727 27826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:15.476275 27826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:33295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:15.477589 27826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:15.478878 27826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:15.490458 27834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:15.490557 27831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:15.490557 27832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:15.491788 27826 server_base.cc:1034] running on GCE node
I20260927 22:15:15.492614 27826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:15.494184 27826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:15.495465 27826 hybrid_clock.cc:648] HybridClock initialized: now 1790547315495369 us; error 74 us; skew 500 ppm
I20260927 22:15:15.495838 27826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:15 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547315, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:15.503818 27826 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:15.505314 27826 webserver.cc:460] Webserver started at http://127.23.173.126:39609/ using document root <none> and password file <none>
I20260927 22:15:15.506083 27826 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:15.506229 27826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:15.506613 27826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:15.509531 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "63adfadf30d541d9a3797b38ea7f752c"
format_stamp: "Formatted at 2026-09-27 22:15:15 on dist-test-slave-3wc3"
server_key: "1854c8c6d970d431879adeb5a570d65c"
server_key_iv: "7687372392bbdf4f88a13fbdfec3575e"
server_key_version: "encryptionkey@0"
I20260927 22:15:15.510362 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "63adfadf30d541d9a3797b38ea7f752c"
format_stamp: "Formatted at 2026-09-27 22:15:15 on dist-test-slave-3wc3"
server_key: "1854c8c6d970d431879adeb5a570d65c"
server_key_iv: "7687372392bbdf4f88a13fbdfec3575e"
server_key_version: "encryptionkey@0"
I20260927 22:15:15.515332 27826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:15:15.518805 27841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:15.520104 27826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:15:15.520289 27826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "63adfadf30d541d9a3797b38ea7f752c"
format_stamp: "Formatted at 2026-09-27 22:15:15 on dist-test-slave-3wc3"
server_key: "1854c8c6d970d431879adeb5a570d65c"
server_key_iv: "7687372392bbdf4f88a13fbdfec3575e"
server_key_version: "encryptionkey@0"
I20260927 22:15:15.520495 27826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:15.550410 27826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:15.554857 27826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:15.555151 27826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:15.573057 27826 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:33295
I20260927 22:15:15.573089 27892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:33295 every 8 connection(s)
I20260927 22:15:15.574442 27826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:15.577927 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27826
I20260927 22:15:15.578217 27893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:15.578226 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:15.578890 24245 external_mini_cluster.cc:1428] Setting key 327ee2ecf35afe1badb0f49f8f5afc76
I20260927 22:15:15.590018 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c: Bootstrap starting.
Sep 27 22:15:15 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:15.593273 27893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:15.594342 27893 log.cc:824] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:15.597362 27893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c: No bootstrap required, opened a new log
I20260927 22:15:15.601452 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:15.580981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.23.173.126:33295)
0927 22:15:15.581712 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:15.581724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:15.581761 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:15:15.584126 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:15.584146 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:15.584536 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:15.584912 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:15.586432 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:15.587784 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:15.589057 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:15.589705 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:15.589904 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:15.594065 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:15.594079 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:15.594093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:15.594132 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:15.597417 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:15.598245 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:15.598250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:15.598256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:15.598415 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:15.598822 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:15.598826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:15.598828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:15.599347 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:15.599593 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:15.600032 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 22:15:15.600412 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":525,"thread_start_us":327,"threads_started":1}
I20260927 22:15:15.605693 27893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } }
I20260927 22:15:15.606071 27893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:15.606180 27893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63adfadf30d541d9a3797b38ea7f752c, State: Initialized, Role: FOLLOWER
I20260927 22:15:15.606706 27893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } }
I20260927 22:15:15.606945 27893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:15.607087 27893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:15.607250 27893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:15.609313 27893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } }
I20260927 22:15:15.609783 27893 leader_election.cc:302] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63adfadf30d541d9a3797b38ea7f752c; no voters:
I20260927 22:15:15.610522 27893 leader_election.cc:288] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:15.610644 27898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:15.611388 27898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [term 1 LEADER]: Becoming Leader. State: Replica: 63adfadf30d541d9a3797b38ea7f752c, State: Running, Role: LEADER
I20260927 22:15:15.611939 27898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } }
I20260927 22:15:15.612638 27893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:15.615237 27900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63adfadf30d541d9a3797b38ea7f752c. Latest consensus state: current_term: 1 leader_uuid: "63adfadf30d541d9a3797b38ea7f752c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } } }
I20260927 22:15:15.615156 27899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63adfadf30d541d9a3797b38ea7f752c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adfadf30d541d9a3797b38ea7f752c" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 33295 } } }
I20260927 22:15:15.615638 27900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:15.615674 27899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:15.619532 27906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:15.624913 27906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:15.634298 27906 catalog_manager.cc:1347] Generated new cluster ID: 32909fc253c647fd8e2a855a2c016bd6
I20260927 22:15:15.634428 27906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:15.655411 27906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:15.657097 27906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:15.668390 27906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63adfadf30d541d9a3797b38ea7f752c: Generated new TSK 0
I20260927 22:15:15.669235 27906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:15.726714 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:16.054648 27919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:16.055083 27919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:16.055256 27919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:16.063664 27919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:16.063776 27919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:16.063889 27919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:16.073668 27919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:16.074977 27919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:16.076210 27919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:16.089495 27924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:16.089428 27927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:16.089464 27925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:16.089937 27919 server_base.cc:1034] running on GCE node
I20260927 22:15:16.090724 27919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:16.091651 27919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:16.092857 27919 hybrid_clock.cc:648] HybridClock initialized: now 1790547316092788 us; error 54 us; skew 500 ppm
I20260927 22:15:16.093175 27919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:16 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:16.096231 27919 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:16.097662 27919 webserver.cc:460] Webserver started at http://127.23.173.65:46805/ using document root <none> and password file <none>
I20260927 22:15:16.098397 27919 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:16.098532 27919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:16.098850 27919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:16.101691 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "cbc5d691e93d4d1ba6f59cf429d09ac3"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "e76a39d562f23329c9342dcb619dfd7a"
server_key_iv: "c0eb140b42b5c45fe3bdcb9ab7961e80"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.102532 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "cbc5d691e93d4d1ba6f59cf429d09ac3"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "e76a39d562f23329c9342dcb619dfd7a"
server_key_iv: "c0eb140b42b5c45fe3bdcb9ab7961e80"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.107577 27919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:15:16.110929 27934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.112267 27919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:15:16.112440 27919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "cbc5d691e93d4d1ba6f59cf429d09ac3"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "e76a39d562f23329c9342dcb619dfd7a"
server_key_iv: "c0eb140b42b5c45fe3bdcb9ab7961e80"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.112654 27919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:16.131466 27919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:16.135710 27919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:16.135996 27919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:16.136955 27919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:16.138381 27919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:16.138533 27919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.138684 27919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:16.138756 27919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.172647 27919 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:37625
I20260927 22:15:16.172704 28046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:37625 every 8 connection(s)
I20260927 22:15:16.174296 27919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:16.175601 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 27919
I20260927 22:15:16.175830 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:16.176651 24245 external_mini_cluster.cc:1428] Setting key cd4013ff48d81903e31e07e14bb7d750
Sep 27 22:15:16 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:16.197911 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:16.177573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:44215 (local address 127.23.173.126:33295)
0927 22:15:16.178124 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:16.178130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:16.180059 (+ 1929us) server_negotiation.cc:407] Connection header received
0927 22:15:16.182995 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:16.183000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:16.183077 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:16.183169 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:16.185292 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.186516 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.188134 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.188571 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.188722 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:16.192310 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:16.192315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:16.192319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:16.192356 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:16.195012 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:16.195727 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:16.195735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:16.195740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:16.195866 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:16.196295 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:16.196302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:16.196305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:16.196628 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:16.196781 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:16.197491 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 22:15:16.197661 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":291,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:15:16.199122 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:16.178266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:33295 (local address 127.23.173.65:44215)
0927 22:15:16.179868 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:16.179928 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:16.182682 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:16.183462 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:16.183477 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:16.184299 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:16.185116 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:16.185143 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.186696 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.186700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:16.187981 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:16.187993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.188815 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.188825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:16.189079 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:16.189793 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:16.189820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:16.192072 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:16.195157 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:16.195166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:16.195185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:16.195608 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:16.195985 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:16.195990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:16.195994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:16.196182 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:16.196754 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:16.196762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:16.197121 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:15:16.197390 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:15:16.197814 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"thread_start_us":339,"threads_started":1}
I20260927 22:15:16.200791 28047 heartbeater.cc:347] Connected to a master server at 127.23.173.126:33295
I20260927 22:15:16.201112 28047 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:16.201829 28047 heartbeater.cc:511] Master 127.23.173.126:33295 requested a full tablet report, sending...
I20260927 22:15:16.204155 27858 ts_manager.cc:194] Registered new tserver with Master: cbc5d691e93d4d1ba6f59cf429d09ac3 (127.23.173.65:37625)
I20260927 22:15:16.205607 27858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:44215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:16.224692 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:16.542923 28053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:16.543182 28053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:16.543390 28053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:16.550434 28053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:16.550536 28053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:16.550733 28053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:16.560349 28053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:16.561645 28053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:16.563035 28053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:16.576184 28059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:16.576428 28058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:16.576884 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:16.577350 28053 server_base.cc:1034] running on GCE node
I20260927 22:15:16.578009 28053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:16.578975 28053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:16.580206 28053 hybrid_clock.cc:648] HybridClock initialized: now 1790547316580153 us; error 43 us; skew 500 ppm
I20260927 22:15:16.580540 28053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:16 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:16.584002 28053 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:16.585556 28053 webserver.cc:460] Webserver started at http://127.23.173.66:44067/ using document root <none> and password file <none>
I20260927 22:15:16.586440 28053 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:16.586573 28053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:16.586874 28053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:16.590690 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "2868a2deed8b4bc7b32588bfae569fa8"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "336d3e546c7c0f98285c4c778bada987"
server_key_iv: "73be02fd04edf97ae36b50c6ac541c0b"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.591648 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "2868a2deed8b4bc7b32588bfae569fa8"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "336d3e546c7c0f98285c4c778bada987"
server_key_iv: "73be02fd04edf97ae36b50c6ac541c0b"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.597033 28053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:15:16.600334 28068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.601639 28053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:16.601763 28053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "2868a2deed8b4bc7b32588bfae569fa8"
format_stamp: "Formatted at 2026-09-27 22:15:16 on dist-test-slave-3wc3"
server_key: "336d3e546c7c0f98285c4c778bada987"
server_key_iv: "73be02fd04edf97ae36b50c6ac541c0b"
server_key_version: "encryptionkey@0"
I20260927 22:15:16.601935 28053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:16.622575 28053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:16.626619 28053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:16.626863 28053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:16.627763 28053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:16.629004 28053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:16.629074 28053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.629158 28053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:16.629227 28053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:16.661936 28053 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34579
I20260927 22:15:16.662001 28180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34579 every 8 connection(s)
I20260927 22:15:16.663267 28053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:16.673390 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28053
I20260927 22:15:16.673591 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:16.674089 24245 external_mini_cluster.cc:1428] Setting key 1947147e465625b20276665da18783ad
Sep 27 22:15:16 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:16.683825 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:16.665860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:53373 (local address 127.23.173.126:33295)
0927 22:15:16.666090 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:16.666096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:16.667281 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:15:16.669255 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:16.669260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:16.669325 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:16.669414 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:16.670886 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.671600 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.673053 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.673457 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.673595 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:16.677239 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:16.677248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:16.677256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:16.677302 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:16.680778 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:16.681469 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:16.681478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:16.681484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:16.681638 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:16.682106 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:16.682111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:16.682113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:16.682503 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:16.682678 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:16.683377 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 22:15:16.683562 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:15:16.684754 28183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:16.666423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:33295 (local address 127.23.173.66:53373)
0927 22:15:16.667117 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:16.667156 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:16.668996 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:16.669625 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:16.669632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:16.670201 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:16.670740 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:16.670757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.671740 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.671743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:16.672932 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:16.672943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:16.673698 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:16.673705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:16.673926 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:16.674697 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:16.674723 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:16.676984 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:16.680913 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:16.680920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:16.680931 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:16.681341 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:16.681741 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:16.681744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:16.681746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:16.681923 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:16.682662 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:16.682669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:16.682972 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 22:15:16.683205 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:15:16.683575 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260927 22:15:16.686210 28181 heartbeater.cc:347] Connected to a master server at 127.23.173.126:33295
I20260927 22:15:16.686556 28181 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:16.687398 28181 heartbeater.cc:511] Master 127.23.173.126:33295 requested a full tablet report, sending...
I20260927 22:15:16.688934 27858 ts_manager.cc:194] Registered new tserver with Master: 2868a2deed8b4bc7b32588bfae569fa8 (127.23.173.66:34579)
I20260927 22:15:16.689740 27858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:53373
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:16.721882 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:17.040819 28186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:17.041131 28186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:17.041292 28186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:17.048839 28186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:17.049018 28186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:17.049199 28186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:17.060051 28186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:33295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:17.061472 28186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:17.062918 28186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:17.075937 28194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:17.075963 28192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:17.075994 28191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:17.077036 28186 server_base.cc:1034] running on GCE node
I20260927 22:15:17.077610 28186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:17.078794 28186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:17.080013 28186 hybrid_clock.cc:648] HybridClock initialized: now 1790547317079963 us; error 44 us; skew 500 ppm
I20260927 22:15:17.080302 28186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:17.083452 28186 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:17.085074 28186 webserver.cc:460] Webserver started at http://127.23.173.67:40427/ using document root <none> and password file <none>
I20260927 22:15:17.085741 28186 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:17.085837 28186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:17.086175 28186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:17.089053 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "766e6ffbe58a49e2969e584c25d60f75"
format_stamp: "Formatted at 2026-09-27 22:15:17 on dist-test-slave-3wc3"
server_key: "a5a24ecabb89ed823df8340534453369"
server_key_iv: "4b9e90966ca7310dac032c7614555afa"
server_key_version: "encryptionkey@0"
I20260927 22:15:17.089772 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "766e6ffbe58a49e2969e584c25d60f75"
format_stamp: "Formatted at 2026-09-27 22:15:17 on dist-test-slave-3wc3"
server_key: "a5a24ecabb89ed823df8340534453369"
server_key_iv: "4b9e90966ca7310dac032c7614555afa"
server_key_version: "encryptionkey@0"
I20260927 22:15:17.094813 28186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:15:17.097947 28201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:17.099290 28186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:17.099411 28186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "766e6ffbe58a49e2969e584c25d60f75"
format_stamp: "Formatted at 2026-09-27 22:15:17 on dist-test-slave-3wc3"
server_key: "a5a24ecabb89ed823df8340534453369"
server_key_iv: "4b9e90966ca7310dac032c7614555afa"
server_key_version: "encryptionkey@0"
I20260927 22:15:17.099586 28186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:17.118813 28186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:17.122540 28186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:17.122766 28186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:17.123613 28186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:17.124963 28186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:17.125036 28186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:17.125120 28186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:17.125165 28186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:17.157815 28186 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:36949
I20260927 22:15:17.157858 28313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:36949 every 8 connection(s)
I20260927 22:15:17.159122 28186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:17.160111 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28186
I20260927 22:15:17.160315 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:17.160797 24245 external_mini_cluster.cc:1428] Setting key 8f8864e091a3c7a817d21e2f1e6f1943
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:17.179925 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.162017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:39591 (local address 127.23.173.126:33295)
0927 22:15:17.162229 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.162234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.163702 (+ 1468us) server_negotiation.cc:407] Connection header received
0927 22:15:17.165694 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.165698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.165763 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.165844 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.167327 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.168066 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.169496 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.169875 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.170020 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.173463 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.173469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.173472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.173539 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.176942 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.177627 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.177632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.177635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.177769 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.178261 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.178266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.178269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.178541 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.178705 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.179373 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.179593 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:15:17.180807 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.162612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:33295 (local address 127.23.173.67:39591)
0927 22:15:17.163529 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:17.163584 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:17.165447 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:17.166103 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:17.166111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:17.166631 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:17.167186 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:17.167203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.168179 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.168183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:17.169382 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:17.169393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.170108 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.170113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:17.170301 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.170938 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:17.170972 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:17.173234 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:17.177073 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:17.177082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:17.177096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:17.177497 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:17.177876 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:17.177882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:17.177886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:17.178145 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:17.178666 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:17.178674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:17.178998 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:15:17.179272 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:15:17.179759 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":347,"threads_started":1}
I20260927 22:15:17.182164 28314 heartbeater.cc:347] Connected to a master server at 127.23.173.126:33295
I20260927 22:15:17.182454 28314 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:17.183149 28314 heartbeater.cc:511] Master 127.23.173.126:33295 requested a full tablet report, sending...
I20260927 22:15:17.184494 27858 ts_manager.cc:194] Registered new tserver with Master: 766e6ffbe58a49e2969e584c25d60f75 (127.23.173.67:36949)
I20260927 22:15:17.185240 27858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:39591
I20260927 22:15:17.188793 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:17.208513 28047 heartbeater.cc:503] Master 127.23.173.126:33295 was elected leader, sending a full tablet report...
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:17.236696 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.222712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.23.173.65:37625)
0927 22:15:17.223342 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.223350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.223408 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:15:17.223489 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.223496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.223747 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.223878 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.225303 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.226388 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.227555 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.228094 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.228188 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.231831 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.231842 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.231856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.231890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.234483 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.235063 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.235071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.235073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.235185 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.235469 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.235472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.235474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.235790 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.236010 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.236216 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.236401 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":324,"threads_started":1}
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:17.253248 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.240790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.23.173.126:33295)
0927 22:15:17.240968 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.240972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.241225 (+ 253us) server_negotiation.cc:407] Connection header received
0927 22:15:17.241343 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.241347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.241409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.241488 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.242786 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.243551 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.244735 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.245120 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.245207 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.248843 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.248848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.248852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.248884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.251227 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.251763 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.251766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.251769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.251885 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.252163 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.252166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.252168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.252451 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.252642 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.252865 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.253017 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 22:15:17.253839 27858 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51264
I20260927 22:15:17.254851 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27919
I20260927 22:15:17.269304 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.255686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.23.173.65:37625)
0927 22:15:17.255841 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.255845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.258348 (+ 2503us) server_negotiation.cc:407] Connection header received
0927 22:15:17.258539 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.258543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.258605 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.258708 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.259870 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.260557 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.261722 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.262206 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.262343 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.264375 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.264380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.264383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.264413 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.266865 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.267464 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.267471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.267475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.267604 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.267953 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.267959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.267963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.268214 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.268465 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.268758 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.268996 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:15:17.269915 28042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:17.351835 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27919
I20260927 22:15:17.367892 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28053
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:17.383297 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.368822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.23.173.66:34579)
0927 22:15:17.369382 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.369390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.369414 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:17.369473 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.369479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.369693 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.369822 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.370981 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.371970 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.373177 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.373692 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.373782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.377794 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.377805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.377819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.377858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.380473 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.381144 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.381152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.381154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.381274 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.381635 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.381641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.381644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.382070 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.382371 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.382678 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.382949 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":301,"threads_started":1}
I20260927 22:15:17.384549 28176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:17.468096 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28053
I20260927 22:15:17.484068 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28186
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[27813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:17.499982 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:17.485077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.23.173.67:36949)
0927 22:15:17.485761 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:17.485768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:17.485796 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:15:17.485858 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:17.485864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:17.486216 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:17.486391 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:17.487669 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.488742 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.490072 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:17.490640 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:17.490776 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:17.494924 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:17.494934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:17.494948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:17.494986 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:17.497600 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.498273 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.498280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.498282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.498399 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:17.498699 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:17.498702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:17.498704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:17.499018 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:17.499252 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:17.499474 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:15:17.499679 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":322,"threads_started":1}
I20260927 22:15:17.501231 28309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:17.583272 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28186
I20260927 22:15:17.598850 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 27826
I20260927 22:15:17.600149 27888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:17.704169 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 27826
2026-09-27T22:15:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3833 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[28333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[28333](info): set up 2 sockets
Sep 27 22:15:17 dist-test-slave-3wc3 krb5kdc[28333](info): commencing operation
krb5kdc: starting...
W20260927 22:15:18.802412 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:18 dist-test-slave-3wc3 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:19.000752 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42041
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:19.330565 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:19.330868 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:19.331040 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:19.339043 28346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:19.339146 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:19.339251 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:19.339332 28346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:19.339419 28346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:19.349725 28346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:19.351070 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:19.352432 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:19.364022 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:19.364022 28354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:19.364163 28351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:19.364825 28346 server_base.cc:1034] running on GCE node
I20260927 22:15:19.365641 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:19.367146 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:19.368388 28346 hybrid_clock.cc:648] HybridClock initialized: now 1790547319368351 us; error 52 us; skew 500 ppm
I20260927 22:15:19.368786 28346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:19 dist-test-slave-3wc3 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547319, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:19.371979 28346 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:19.373199 28346 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:19.373337 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:19.373694 28346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:19.376758 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "7ef0c359bb634d188088def928973c13"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "f748bc09882b86d0bbeb624075acb82b"
server_key_iv: "36c36032b668c2823eaa3850c212845d"
server_key_version: "encryptionkey@0"
I20260927 22:15:19.377565 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "7ef0c359bb634d188088def928973c13"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "f748bc09882b86d0bbeb624075acb82b"
server_key_iv: "36c36032b668c2823eaa3850c212845d"
server_key_version: "encryptionkey@0"
I20260927 22:15:19.382576 28346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:15:19.386030 28360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:19.387418 28346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:19.387614 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "7ef0c359bb634d188088def928973c13"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "f748bc09882b86d0bbeb624075acb82b"
server_key_iv: "36c36032b668c2823eaa3850c212845d"
server_key_version: "encryptionkey@0"
I20260927 22:15:19.387835 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:19.430400 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:19.435549 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:19.435914 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:19.456379 28346 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42041
I20260927 22:15:19.456351 28411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42041 every 8 connection(s)
I20260927 22:15:19.457408 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:19.458423 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28346
I20260927 22:15:19.458667 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:19.459443 24245 external_mini_cluster.cc:1428] Setting key dd629623a201acfa91c1486a5f869201
I20260927 22:15:19.462311 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:15:19 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:19.475423 28412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: Bootstrap starting.
I20260927 22:15:19.478538 28412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:19.479573 28412 log.cc:824] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:19.482370 28412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: No bootstrap required, opened a new log
I20260927 22:15:19.483264 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:19.461696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.23.173.126:42041)
0927 22:15:19.462741 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:19.462754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:19.462798 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:15:19.464532 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:19.464554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:19.464972 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:19.465327 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:19.467181 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:19.468660 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:19.470473 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:19.471152 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:19.471300 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:19.476071 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:19.476079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:19.476093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:19.476128 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:19.479261 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:19.480014 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:19.480020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:19.480025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:19.480190 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:19.480590 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:19.480594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:19.480596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:19.481098 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:19.481408 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:19.481839 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 22:15:19.482280 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":590,"threads_started":1}
I20260927 22:15:19.490173 28412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } }
I20260927 22:15:19.490536 28412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:19.490643 28412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0c359bb634d188088def928973c13, State: Initialized, Role: FOLLOWER
I20260927 22:15:19.491247 28412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } }
I20260927 22:15:19.491515 28412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:19.491672 28412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:19.491863 28412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:19.494311 28412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } }
I20260927 22:15:19.494840 28412 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef0c359bb634d188088def928973c13; no voters:
I20260927 22:15:19.495378 28412 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:19.495530 28417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:19.496320 28417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [term 1 LEADER]: Becoming Leader. State: Replica: 7ef0c359bb634d188088def928973c13, State: Running, Role: LEADER
I20260927 22:15:19.496901 28417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } }
I20260927 22:15:19.497716 28412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:19.500116 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef0c359bb634d188088def928973c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } } }
I20260927 22:15:19.500454 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:19.500154 28419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef0c359bb634d188088def928973c13. Latest consensus state: current_term: 1 leader_uuid: "7ef0c359bb634d188088def928973c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef0c359bb634d188088def928973c13" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42041 } } }
I20260927 22:15:19.500844 28419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:19.502166 28424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:19.507215 28424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:19.517459 28424 catalog_manager.cc:1347] Generated new cluster ID: 692fbf0e09b74ecabea0fcf6cddc6699
I20260927 22:15:19.517580 28424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:19.553725 28424 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:19.555133 28424 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:19.562908 28424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: Generated new TSK 0
I20260927 22:15:19.563987 28424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:19.620923 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:19.944264 28438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:19.944603 28438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:19.944831 28438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:19.952128 28438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:19.952224 28438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:19.952366 28438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:19.962040 28438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:19.963346 28438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:19.964670 28438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:19.977409 28443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:19.977416 28444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:19.977437 28446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:19.978338 28438 server_base.cc:1034] running on GCE node
I20260927 22:15:19.978912 28438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:19.979838 28438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:19.981031 28438 hybrid_clock.cc:648] HybridClock initialized: now 1790547319980984 us; error 39 us; skew 500 ppm
I20260927 22:15:19.981372 28438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:19 dist-test-slave-3wc3 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547319, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:19.984620 28438 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:19.985790 28438 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:19.985879 28438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:19.986204 28438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:19.989099 28438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "f10a6a07eb6a4507bdbb55bdc39320f3"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "c921da376324e895d78f50343e615175"
server_key_iv: "82731136c8e6ec7e694f551284a3383e"
server_key_version: "encryptionkey@0"
I20260927 22:15:19.989849 28438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "f10a6a07eb6a4507bdbb55bdc39320f3"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "c921da376324e895d78f50343e615175"
server_key_iv: "82731136c8e6ec7e694f551284a3383e"
server_key_version: "encryptionkey@0"
I20260927 22:15:19.994897 28438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:15:19.998242 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:19.999639 28438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:15:19.999795 28438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "f10a6a07eb6a4507bdbb55bdc39320f3"
format_stamp: "Formatted at 2026-09-27 22:15:19 on dist-test-slave-3wc3"
server_key: "c921da376324e895d78f50343e615175"
server_key_iv: "82731136c8e6ec7e694f551284a3383e"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.000016 28438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:20.016009 28438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:20.020314 28438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:20.020593 28438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:20.021512 28438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:20.022903 28438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:20.022990 28438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.023145 28438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:20.023247 28438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.066569 28438 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40313
I20260927 22:15:20.066645 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40313 every 8 connection(s)
I20260927 22:15:20.067437 28438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:20.070029 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28438
I20260927 22:15:20.070241 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:20.071053 24245 external_mini_cluster.cc:1428] Setting key e30bf01d490ec2bffda57a1e144b7b5f
Sep 27 22:15:20 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547319, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:20.087425 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:20.069897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:36385 (local address 127.23.173.126:42041)
0927 22:15:20.070365 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:20.070370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:20.071619 (+ 1249us) server_negotiation.cc:407] Connection header received
0927 22:15:20.073426 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:20.073432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:20.073524 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:20.073640 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:20.075190 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.075890 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.077411 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.077803 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.077935 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:20.081803 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:20.081808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:20.081811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:20.081841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:20.084404 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:20.085130 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:20.085139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:20.085142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:20.085253 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:20.085671 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:20.085676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:20.085679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:20.085925 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:20.086129 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:20.086797 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 22:15:20.087032 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":269,"threads_started":1}
I20260927 22:15:20.088356 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:20.070644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42041 (local address 127.23.173.65:36385)
0927 22:15:20.071453 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:20.071492 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:20.073166 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:20.073870 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:20.073879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:20.074451 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:20.075021 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:20.075043 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.076049 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.076059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:20.077264 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:20.077276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.078048 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.078055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:20.078281 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:20.078950 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:20.078983 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:20.081581 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:20.084547 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:20.084555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:20.084572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:20.085001 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:20.085360 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:20.085365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:20.085369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:20.085553 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:20.086113 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:20.086120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:20.086438 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 22:15:20.086699 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 22:15:20.087253 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:20.090176 28565 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42041
I20260927 22:15:20.090574 28565 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:20.091313 28565 heartbeater.cc:511] Master 127.23.173.126:42041 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:15:20.093508 28377 ts_manager.cc:194] Registered new tserver with Master: f10a6a07eb6a4507bdbb55bdc39320f3 (127.23.173.65:40313)
I20260927 22:15:20.095014 28377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:36385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:20.115132 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:20.433473 28571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:20.433799 28571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:20.433902 28571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:20.441834 28571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:20.442025 28571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:20.442173 28571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:20.452220 28571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:20.453461 28571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:20.454826 28571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:20.467985 28577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:20.468187 28576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:20.468466 28579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:20.468695 28571 server_base.cc:1034] running on GCE node
I20260927 22:15:20.469287 28571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:20.470253 28571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:20.471515 28571 hybrid_clock.cc:648] HybridClock initialized: now 1790547320471440 us; error 42 us; skew 500 ppm
I20260927 22:15:20.471832 28571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:20 dist-test-slave-3wc3 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:20.475064 28571 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:20.476260 28571 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:20.476393 28571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:20.476704 28571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:20.479791 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "063cea88a79346e09b575d1b85f35e3e"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "d54f2303e6fc7656a1bc9be1ffe86fc8"
server_key_iv: "cc4c36de1ca6cbbb18c30194887a44fa"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.480614 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "063cea88a79346e09b575d1b85f35e3e"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "d54f2303e6fc7656a1bc9be1ffe86fc8"
server_key_iv: "cc4c36de1ca6cbbb18c30194887a44fa"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.485342 28571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 22:15:20.488616 28585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.489841 28571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:20.490052 28571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "063cea88a79346e09b575d1b85f35e3e"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "d54f2303e6fc7656a1bc9be1ffe86fc8"
server_key_iv: "cc4c36de1ca6cbbb18c30194887a44fa"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.490295 28571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:20.519258 28571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:20.523453 28571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:20.523739 28571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:20.524669 28571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:20.525930 28571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:20.526114 28571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.526237 28571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:20.526326 28571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.558542 28571 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:42581
I20260927 22:15:20.558633 28697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:42581 every 8 connection(s)
I20260927 22:15:20.559420 28571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:20.563832 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28571
I20260927 22:15:20.564036 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:20.564574 24245 external_mini_cluster.cc:1428] Setting key ff650929ccd65c7c8b96b1cbd5c245e2
Sep 27 22:15:20 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:20.581224 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:20.561835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:48723 (local address 127.23.173.126:42041)
0927 22:15:20.562058 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:20.562063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:20.563373 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 22:15:20.565799 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:20.565805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:20.565875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:20.566004 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:20.567936 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.568609 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.571110 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.571524 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.571620 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:20.575488 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:20.575492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:20.575496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:20.575526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:20.578061 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:20.578980 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:20.578983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:20.578986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:20.579104 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:20.579594 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:20.579597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:20.579599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:20.579842 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:20.579947 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:20.580811 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 22:15:20.580975 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260927 22:15:20.582571 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:20.562400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42041 (local address 127.23.173.66:48723)
0927 22:15:20.563171 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:20.563232 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:20.565507 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:20.566220 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:20.566228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:20.566969 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:20.567760 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:20.567788 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.568759 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.568767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:20.570959 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:20.570975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:20.571741 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:20.571750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:20.572057 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:20.572780 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:20.572814 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:20.575280 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:20.578191 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:20.578202 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:20.578222 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:20.578869 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:20.579211 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:20.579218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:20.579224 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:20.579474 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:20.579966 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:20.579976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:20.580417 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 22:15:20.580731 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 22:15:20.581295 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":284,"threads_started":1}
I20260927 22:15:20.584002 28698 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42041
I20260927 22:15:20.584336 28698 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:20.585137 28698 heartbeater.cc:511] Master 127.23.173.126:42041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:15:20.586735 28377 ts_manager.cc:194] Registered new tserver with Master: 063cea88a79346e09b575d1b85f35e3e (127.23.173.66:42581)
I20260927 22:15:20.587568 28377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:48723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:20.612461 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:20.931672 28703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:20.931936 28703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:20.932032 28703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:20.938998 28703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:20.939100 28703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:20.939208 28703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:20.948490 28703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:20.949864 28703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:20.951152 28703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:20.963398 28711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:20.963490 28709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:20.963675 28708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:20.964614 28703 server_base.cc:1034] running on GCE node
I20260927 22:15:20.965173 28703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:20.966084 28703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:20.967272 28703 hybrid_clock.cc:648] HybridClock initialized: now 1790547320967219 us; error 54 us; skew 500 ppm
I20260927 22:15:20.967566 28703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:20 dist-test-slave-3wc3 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:20.970664 28703 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:20.971824 28703 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:20.971958 28703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:20.972266 28703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:20.975337 28703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "5de14eecf9b64d219d1d9b75646d65d2"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "23a3f11835a717518a5647ac57e46b8d"
server_key_iv: "837b7b255c7637f844d7ecc340470942"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.976146 28703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "5de14eecf9b64d219d1d9b75646d65d2"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "23a3f11835a717518a5647ac57e46b8d"
server_key_iv: "837b7b255c7637f844d7ecc340470942"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.980978 28703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:15:20.984272 28717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:20.985512 28703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:20.985674 28703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "5de14eecf9b64d219d1d9b75646d65d2"
format_stamp: "Formatted at 2026-09-27 22:15:20 on dist-test-slave-3wc3"
server_key: "23a3f11835a717518a5647ac57e46b8d"
server_key_iv: "837b7b255c7637f844d7ecc340470942"
server_key_version: "encryptionkey@0"
I20260927 22:15:20.985882 28703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:21.029022 28703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:21.033483 28703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:21.033774 28703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:21.034780 28703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:21.036077 28703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:21.036197 28703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:21.036329 28703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:21.036419 28703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:21.069265 28703 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:42379
I20260927 22:15:21.069340 28829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:42379 every 8 connection(s)
I20260927 22:15:21.070322 28703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:21.071054 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28703
I20260927 22:15:21.071272 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:21.071820 24245 external_mini_cluster.cc:1428] Setting key 0989db321f8d3d7ba07c6d867dce41a7
Sep 27 22:15:21 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547320, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:21.090978 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.074200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:35305 (local address 127.23.173.126:42041)
0927 22:15:21.074445 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.074454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.075626 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 22:15:21.077685 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.077689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.077752 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.077835 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:21.079312 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.080029 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.081453 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.081836 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.082013 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.085222 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:21.085227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:21.085230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:21.085260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:21.088266 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.088979 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.088983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.088986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.089120 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.089473 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.089476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.089478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.089776 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:21.089903 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.090578 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.090728 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:15:21.091938 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.073720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42041 (local address 127.23.173.67:35305)
0927 22:15:21.075452 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.075502 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.077454 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.078146 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.078154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.078656 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:21.079170 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.079186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.080164 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.080167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.081333 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.081343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.082090 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.082096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.082271 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.082839 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:21.082859 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:21.085037 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:21.088384 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:21.088389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:21.088400 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:21.088796 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:21.089212 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:21.089215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:21.089218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:21.089377 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:21.089892 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:21.089897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:21.090257 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.090472 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.090806 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452,"thread_start_us":1218,"threads_started":1}
I20260927 22:15:21.093138 28830 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42041
I20260927 22:15:21.093466 28830 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:21.094187 28830 heartbeater.cc:511] Master 127.23.173.126:42041 requested a full tablet report, sending...
I20260927 22:15:21.095496 28377 ts_manager.cc:194] Registered new tserver with Master: 5de14eecf9b64d219d1d9b75646d65d2 (127.23.173.67:42379)
I20260927 22:15:21.096259 28377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:35305
I20260927 22:15:21.097994 28565 heartbeater.cc:503] Master 127.23.173.126:42041 was elected leader, sending a full tablet report...
I20260927 22:15:21.099792 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:21.116279 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.104253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.23.173.126:42041)
0927 22:15:21.104419 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.104427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.104938 (+ 511us) server_negotiation.cc:407] Connection header received
0927 22:15:21.105124 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.105129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.105192 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.105290 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:21.106768 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.107466 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.108726 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.109148 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.109277 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.111430 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:21.111435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:21.111438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:21.111466 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:21.113761 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.114427 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.114432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.114436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.114561 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.114908 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.114913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.114917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.115206 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:21.115465 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.115765 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.115970 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:15:21.120019 28377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:21.126770 28377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:21.150003 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.138713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:42379 (local address 127.0.0.1:53886)
0927 22:15:21.141357 (+ 2644us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.141370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.141520 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.142479 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.142486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.142512 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:21.142815 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.142822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.144733 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.144741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.147611 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.147621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.148835 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.148841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.149029 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.149082 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.149252 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.149456 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2546,"thread_start_us":317,"threads_started":1}
I20260927 22:15:21.150135 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.139385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.23.173.67:42379)
0927 22:15:21.140101 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.140112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.141652 (+ 1540us) server_negotiation.cc:407] Connection header received
0927 22:15:21.141751 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.141759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.142031 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.142206 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:21.142996 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.144551 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.147752 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.148654 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.148805 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.149046 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.149591 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.149778 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":408,"threads_started":1}
I20260927 22:15:21.151130 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.138459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40313 (local address 127.0.0.1:50906)
0927 22:15:21.139436 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.139487 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.139656 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.140959 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.140970 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.141013 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:21.141499 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.141512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.145116 (+ 3604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.145124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.147060 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.147072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.149291 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.149297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.149474 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.150509 (+ 1035us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.150614 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.150739 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":987,"thread_start_us":2683,"threads_started":1}
I20260927 22:15:21.151183 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.138911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.23.173.65:40313)
0927 22:15:21.140452 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.140459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.140480 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:15:21.140553 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.140558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.140733 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.140872 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:21.141720 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.144550 (+ 2830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.147221 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.148731 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.148868 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.149136 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.150689 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.150875 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":1272,"threads_started":1}
I20260927 22:15:21.153641 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.141566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42581 (local address 127.0.0.1:49962)
0927 22:15:21.145612 (+ 4046us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.145629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.145753 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.146704 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.146709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.146732 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:21.147172 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.147182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.149028 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.149033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.151228 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.151247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.152892 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.152898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.153061 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.153107 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.153208 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.153322 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3915,"thread_start_us":3730,"threads_started":1}
I20260927 22:15:21.153666 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.142093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.23.173.66:42581)
0927 22:15:21.145124 (+ 3031us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.145132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.145883 (+ 751us) server_negotiation.cc:407] Connection header received
0927 22:15:21.146071 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.146080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.146296 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.146448 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:21.147361 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.148878 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.151403 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.152692 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.152834 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.153020 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.153248 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.153392 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2867,"thread_start_us":2744,"threads_started":1}
I20260927 22:15:21.154367 28765 tablet_service.cc:1467] Processing CreateTablet for tablet b156726572e248ceaaabbac40046ed39 (DEFAULT_TABLE table=test-table [id=d6858265a49e43f292883b168f0cb16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:21.154980 28500 tablet_service.cc:1467] Processing CreateTablet for tablet b156726572e248ceaaabbac40046ed39 (DEFAULT_TABLE table=test-table [id=d6858265a49e43f292883b168f0cb16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:21.155898 28765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156726572e248ceaaabbac40046ed39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:21.156306 28500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156726572e248ceaaabbac40046ed39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:21.156791 28633 tablet_service.cc:1467] Processing CreateTablet for tablet b156726572e248ceaaabbac40046ed39 (DEFAULT_TABLE table=test-table [id=d6858265a49e43f292883b168f0cb16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:21.157780 28633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b156726572e248ceaaabbac40046ed39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:21.168560 28848 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Bootstrap starting.
I20260927 22:15:21.171818 28848 tablet_bootstrap.cc:654] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:21.172051 28849 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Bootstrap starting.
I20260927 22:15:21.173177 28850 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Bootstrap starting.
I20260927 22:15:21.173173 28848 log.cc:824] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:21.176524 28850 tablet_bootstrap.cc:654] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:21.176801 28849 tablet_bootstrap.cc:654] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:21.177603 28850 log.cc:824] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:21.178256 28849 log.cc:824] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:21.180485 28848 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: No bootstrap required, opened a new log
I20260927 22:15:21.180562 28850 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: No bootstrap required, opened a new log
I20260927 22:15:21.180778 28848 ts_tablet_manager.cc:1397] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260927 22:15:21.180861 28850 ts_tablet_manager.cc:1397] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 22:15:21.181193 28849 tablet_bootstrap.cc:492] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: No bootstrap required, opened a new log
I20260927 22:15:21.181434 28849 ts_tablet_manager.cc:1397] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 22:15:21.187965 28848 raft_consensus.cc:348] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.188329 28848 raft_consensus.cc:374] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:21.188450 28848 raft_consensus.cc:729] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10a6a07eb6a4507bdbb55bdc39320f3, State: Initialized, Role: FOLLOWER
I20260927 22:15:21.189155 28848 consensus_queue.cc:260] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.191370 28848 ts_tablet_manager.cc:1428] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Time spent starting tablet: real 0.010s user 0.010s sys 0.002s
I20260927 22:15:21.191298 28850 raft_consensus.cc:348] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.191744 28850 raft_consensus.cc:374] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:21.191855 28850 raft_consensus.cc:729] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063cea88a79346e09b575d1b85f35e3e, State: Initialized, Role: FOLLOWER
I20260927 22:15:21.191808 28849 raft_consensus.cc:348] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.192201 28849 raft_consensus.cc:374] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:21.192303 28849 raft_consensus.cc:729] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de14eecf9b64d219d1d9b75646d65d2, State: Initialized, Role: FOLLOWER
I20260927 22:15:21.192605 28850 consensus_queue.cc:260] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.192942 28849 consensus_queue.cc:260] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.193660 28698 heartbeater.cc:503] Master 127.23.173.126:42041 was elected leader, sending a full tablet report...
I20260927 22:15:21.194591 28850 ts_tablet_manager.cc:1428] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260927 22:15:21.195784 28830 heartbeater.cc:503] Master 127.23.173.126:42041 was elected leader, sending a full tablet report...
I20260927 22:15:21.196214 28849 ts_tablet_manager.cc:1428] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
W20260927 22:15:21.311424 28699 tablet.cc:2369] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:21.319674 28566 tablet.cc:2369] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:21.326848 28831 tablet.cc:2369] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:15:21.590021 28856 raft_consensus.cc:482] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:21.590344 28856 raft_consensus.cc:504] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.591977 28856 leader_election.cc:288] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10a6a07eb6a4507bdbb55bdc39320f3 (127.23.173.65:40313), 063cea88a79346e09b575d1b85f35e3e (127.23.173.66:42581)
I20260927 22:15:21.598441 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.592389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:57321 (local address 127.23.173.65:40313)
0927 22:15:21.592609 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.592617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.592640 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:15:21.592742 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.592747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.592827 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.592937 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:21.593514 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.594601 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.596476 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.597582 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.597693 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.597775 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.598126 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.598214 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:15:21.598556 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.592301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40313 (local address 127.23.173.67:57321)
0927 22:15:21.592468 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.592495 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.592595 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.593126 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.593131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.593165 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:21.593339 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.593347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.594724 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.594731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.596322 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.596334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.597740 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.597746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.597934 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.598000 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.598103 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.598240 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":19}
I20260927 22:15:21.599470 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.592964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:42435 (local address 127.23.173.66:42581)
0927 22:15:21.593196 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.593201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.593306 (+ 105us) server_negotiation.cc:407] Connection header received
0927 22:15:21.593540 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.593549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.593607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.593674 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:21.594328 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.595370 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.597359 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.598454 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.598620 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.598700 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.599196 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.599300 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:15:21.599658 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.592720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42581 (local address 127.23.173.67:42435)
0927 22:15:21.593201 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:21.593222 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:21.593387 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:21.593722 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:21.593726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:21.593752 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:21.594135 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.594147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.595523 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.595526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.597202 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:21.597218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.598849 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.598858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:21.598999 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.599022 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:15:21.599150 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 22:15:21.599285 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":244,"threads_started":1}
I20260927 22:15:21.599721 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b156726572e248ceaaabbac40046ed39" candidate_uuid: "5de14eecf9b64d219d1d9b75646d65d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" is_pre_election: true
I20260927 22:15:21.600308 28520 raft_consensus.cc:2454] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de14eecf9b64d219d1d9b75646d65d2 in term 0.
I20260927 22:15:21.600641 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b156726572e248ceaaabbac40046ed39" candidate_uuid: "5de14eecf9b64d219d1d9b75646d65d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063cea88a79346e09b575d1b85f35e3e" is_pre_election: true
I20260927 22:15:21.601027 28719 leader_election.cc:302] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de14eecf9b64d219d1d9b75646d65d2, f10a6a07eb6a4507bdbb55bdc39320f3; no voters:
I20260927 22:15:21.601176 28653 raft_consensus.cc:2454] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de14eecf9b64d219d1d9b75646d65d2 in term 0.
I20260927 22:15:21.601389 28856 raft_consensus.cc:2781] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:21.601529 28856 raft_consensus.cc:482] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:21.601691 28856 raft_consensus.cc:3037] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:21.604063 28856 raft_consensus.cc:504] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.604640 28856 leader_election.cc:288] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [CANDIDATE]: Term 1 election: Requested vote from peers f10a6a07eb6a4507bdbb55bdc39320f3 (127.23.173.65:40313), 063cea88a79346e09b575d1b85f35e3e (127.23.173.66:42581)
I20260927 22:15:21.605100 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b156726572e248ceaaabbac40046ed39" candidate_uuid: "5de14eecf9b64d219d1d9b75646d65d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3"
I20260927 22:15:21.605305 28520 raft_consensus.cc:3037] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:21.605269 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b156726572e248ceaaabbac40046ed39" candidate_uuid: "5de14eecf9b64d219d1d9b75646d65d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063cea88a79346e09b575d1b85f35e3e"
I20260927 22:15:21.605492 28653 raft_consensus.cc:3037] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:21.607542 28520 raft_consensus.cc:2454] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de14eecf9b64d219d1d9b75646d65d2 in term 1.
I20260927 22:15:21.607977 28653 raft_consensus.cc:2454] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de14eecf9b64d219d1d9b75646d65d2 in term 1.
I20260927 22:15:21.607978 28719 leader_election.cc:302] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5de14eecf9b64d219d1d9b75646d65d2, f10a6a07eb6a4507bdbb55bdc39320f3; no voters:
I20260927 22:15:21.608297 28856 raft_consensus.cc:2781] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:21.608750 28856 raft_consensus.cc:686] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5de14eecf9b64d219d1d9b75646d65d2, State: Running, Role: LEADER
I20260927 22:15:21.609310 28856 consensus_queue.cc:237] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } }
I20260927 22:15:21.613875 28376 catalog_manager.cc:5495] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de14eecf9b64d219d1d9b75646d65d2 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "5de14eecf9b64d219d1d9b75646d65d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de14eecf9b64d219d1d9b75646d65d2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:21.711855 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.705841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.23.173.67:42379)
0927 22:15:21.706355 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.706363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.706582 (+ 219us) server_negotiation.cc:407] Connection header received
0927 22:15:21.706848 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.706852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.706930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.707116 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:21.707592 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.708257 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.709660 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.710153 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.710286 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.710757 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:21.711242 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:21.711414 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.711472 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.711582 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260927 22:15:21.719664 28653 raft_consensus.cc:1261] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 1 FOLLOWER]: Refusing update from remote peer 5de14eecf9b64d219d1d9b75646d65d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:21.719702 28520 raft_consensus.cc:1261] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 1 FOLLOWER]: Refusing update from remote peer 5de14eecf9b64d219d1d9b75646d65d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:21.720643 28858 consensus_queue.cc:1035] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063cea88a79346e09b575d1b85f35e3e" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:21.721068 28856 consensus_queue.cc:1035] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10a6a07eb6a4507bdbb55bdc39320f3" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:21.732328 28863 mvcc.cc:205] Tried to move back new op lower bound from 7334081829754404864 to 7334081829316120576. Current Snapshot: MvccSnapshot[applied={T|T < 7334081829754404864}]
I20260927 22:15:21.745774 28376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:21.746088 28376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:21.749426 28376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ef0c359bb634d188088def928973c13: Sending DeleteTablet for 3 replicas of tablet b156726572e248ceaaabbac40046ed39
I20260927 22:15:21.750686 28633 tablet_service.cc:1514] Processing DeleteTablet for tablet b156726572e248ceaaabbac40046ed39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:21 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:49962
I20260927 22:15:21.750694 28765 tablet_service.cc:1514] Processing DeleteTablet for tablet b156726572e248ceaaabbac40046ed39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:21 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:53886
I20260927 22:15:21.750713 28500 tablet_service.cc:1514] Processing DeleteTablet for tablet b156726572e248ceaaabbac40046ed39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:21 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:50906
I20260927 22:15:21.755175 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28438
I20260927 22:15:21.765486 28875 tablet_replica.cc:330] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: stopping tablet replica
I20260927 22:15:21.766083 28875 raft_consensus.cc:2229] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:21.766690 28875 raft_consensus.cc:2258] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:21.766752 28876 tablet_replica.cc:330] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: stopping tablet replica
I20260927 22:15:21.767187 28876 raft_consensus.cc:2229] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:21.767668 28876 raft_consensus.cc:2258] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:21.769027 28875 ts_tablet_manager.cc:1905] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:21.769588 28876 ts_tablet_manager.cc:1905] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:21.776885 28875 ts_tablet_manager.cc:1918] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:21.777057 28875 log.cc:1196] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/b156726572e248ceaaabbac40046ed39
I20260927 22:15:21.777084 28876 ts_tablet_manager.cc:1918] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:21.777243 28876 log.cc:1196] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/b156726572e248ceaaabbac40046ed39
I20260927 22:15:21.777514 28875 ts_tablet_manager.cc:1939] T b156726572e248ceaaabbac40046ed39 P 063cea88a79346e09b575d1b85f35e3e: Deleting consensus metadata
I20260927 22:15:21.777750 28876 ts_tablet_manager.cc:1939] T b156726572e248ceaaabbac40046ed39 P f10a6a07eb6a4507bdbb55bdc39320f3: Deleting consensus metadata
I20260927 22:15:21.779366 28362 catalog_manager.cc:4841] TS 063cea88a79346e09b575d1b85f35e3e (127.23.173.66:42581): tablet b156726572e248ceaaabbac40046ed39 (table test-table [id=d6858265a49e43f292883b168f0cb16d]) successfully deleted
I20260927 22:15:21.779800 28362 catalog_manager.cc:4841] TS f10a6a07eb6a4507bdbb55bdc39320f3 (127.23.173.65:40313): tablet b156726572e248ceaaabbac40046ed39 (table test-table [id=d6858265a49e43f292883b168f0cb16d]) successfully deleted
I20260927 22:15:21.780478 28877 tablet_replica.cc:330] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: stopping tablet replica
I20260927 22:15:21.780894 28877 raft_consensus.cc:2229] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:21.781481 28877 raft_consensus.cc:2258] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:21.783020 28877 ts_tablet_manager.cc:1905] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:15:21 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:21.788904 28877 ts_tablet_manager.cc:1918] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:21.789081 28877 log.cc:1196] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/b156726572e248ceaaabbac40046ed39
I20260927 22:15:21.789619 28877 ts_tablet_manager.cc:1939] T b156726572e248ceaaabbac40046ed39 P 5de14eecf9b64d219d1d9b75646d65d2: Deleting consensus metadata
I20260927 22:15:21.791287 28361 catalog_manager.cc:4841] TS 5de14eecf9b64d219d1d9b75646d65d2 (127.23.173.67:42379): tablet b156726572e248ceaaabbac40046ed39 (table test-table [id=d6858265a49e43f292883b168f0cb16d]) successfully deleted
I20260927 22:15:21.793490 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.757037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.23.173.65:40313)
0927 22:15:21.757290 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.757299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.774005 (+ 16706us) server_negotiation.cc:407] Connection header received
0927 22:15:21.774506 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.774512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.774599 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.774674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:21.776413 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.777605 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.780232 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.780922 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.781025 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.786374 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:21.786388 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:21.786407 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:21.786447 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:21.790316 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.791189 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.791196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.791198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.791389 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.791804 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.791810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.791813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.792270 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:21.792618 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.792886 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.793110 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:15:21.794206 28560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:21.882794 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28438
I20260927 22:15:21.901718 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28571
Sep 27 22:15:21 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:21.918385 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:21.902594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.23.173.66:42581)
0927 22:15:21.902779 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:21.902787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:21.902851 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:15:21.903067 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:21.903071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:21.903145 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:21.903227 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:21.904763 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.905480 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.907233 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:21.907621 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:21.907710 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:21.912647 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:21.912662 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:21.912677 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:21.912710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:21.915484 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.916223 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.916229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.916232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.916355 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:21.916729 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:21.916732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:21.916734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:21.917092 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:21.917305 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:21.917636 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:15:21.917794 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:15:21.919097 28693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:22.010201 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28571
I20260927 22:15:22.027338 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28703
Sep 27 22:15:22 dist-test-slave-3wc3 krb5kdc[28333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:22.043213 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:22.028262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.23.173.67:42379)
0927 22:15:22.028459 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:22.028471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:22.028493 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:15:22.028576 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:22.028582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:22.028658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:22.028777 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:22.030057 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:22.030842 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:22.032091 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:22.032632 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:22.032766 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:22.037305 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:22.037319 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:22.037334 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:22.037372 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:22.040397 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:22.041026 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:22.041040 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:22.041042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:22.041175 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:22.041547 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:22.041558 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:22.041560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:22.041946 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:22.042330 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:22.042634 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:15:22.042901 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:15:22.044050 28825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:22.166019 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28703
I20260927 22:15:22.185058 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28346
I20260927 22:15:22.186254 28407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:22.294898 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28346
2026-09-27T22:15:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4590 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:22Z Disabled control of system clock
I20260927 22:15:22.360833 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40703
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:15:22.678277 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:22.678574 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:22.678664 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:22.686755 28886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:22.686859 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:22.686900 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:22.686965 28886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:22.687013 28886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:22.696587 28886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:40703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:22.697805 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:22.699550 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:22.711015 28894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:22.711144 28892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:22.711155 28891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:22.712091 28886 server_base.cc:1034] running on GCE node
I20260927 22:15:22.712857 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:22.714285 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:22.715480 28886 hybrid_clock.cc:648] HybridClock initialized: now 1790547322715445 us; error 35 us; skew 500 ppm
I20260927 22:15:22.715833 28886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:22.718215 28886 webserver.cc:460] Webserver started at http://127.23.173.126:44945/ using document root <none> and password file <none>
I20260927 22:15:22.718892 28886 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:22.719004 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:22.719344 28886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:22.722281 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "b6987c4a92a844fe846d6c43319248bc"
format_stamp: "Formatted at 2026-09-27 22:15:22 on dist-test-slave-3wc3"
server_key: "aaf6c2c793feb4061429223a8f2ca038"
server_key_iv: "d6b4fe594ab05ecff084431462bb5328"
server_key_version: "encryptionkey@0"
I20260927 22:15:22.723069 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "b6987c4a92a844fe846d6c43319248bc"
format_stamp: "Formatted at 2026-09-27 22:15:22 on dist-test-slave-3wc3"
server_key: "aaf6c2c793feb4061429223a8f2ca038"
server_key_iv: "d6b4fe594ab05ecff084431462bb5328"
server_key_version: "encryptionkey@0"
I20260927 22:15:22.728107 28886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:22.731484 28900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:22.732736 28886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 22:15:22.732923 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "b6987c4a92a844fe846d6c43319248bc"
format_stamp: "Formatted at 2026-09-27 22:15:22 on dist-test-slave-3wc3"
server_key: "aaf6c2c793feb4061429223a8f2ca038"
server_key_iv: "d6b4fe594ab05ecff084431462bb5328"
server_key_version: "encryptionkey@0"
I20260927 22:15:22.733146 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:22.738763 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:22.739504 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:22.739794 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:22.757699 28886 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:40703
I20260927 22:15:22.757722 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:40703 every 8 connection(s)
I20260927 22:15:22.759100 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:22.762847 28952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:22.768003 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28886
I20260927 22:15:22.768204 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:22.768827 24245 external_mini_cluster.cc:1428] Setting key 80dce8edb9d49e2c3e030810a5068a12
I20260927 22:15:22.774246 28952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: Bootstrap starting.
I20260927 22:15:22.778167 28952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:22.778291 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:22.770875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.23.173.126:40703)
0927 22:15:22.771994 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:22.772009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:22.772060 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:15:22.774361 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:22.774389 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:22.774892 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:22.775312 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:22.776365 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:22.776374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:22.776390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:22.776409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:22.776492 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:22.776514 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:22.776716 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:22.776835 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:15:22.777095 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":371,"threads_started":1}
I20260927 22:15:22.779300 28952 log.cc:824] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:22.782259 28952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: No bootstrap required, opened a new log
I20260927 22:15:22.789620 28952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } }
I20260927 22:15:22.790024 28952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:22.790141 28952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6987c4a92a844fe846d6c43319248bc, State: Initialized, Role: FOLLOWER
I20260927 22:15:22.790841 28952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } }
I20260927 22:15:22.791105 28952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:22.791256 28952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:22.791437 28952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:22.793553 28952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } }
I20260927 22:15:22.794131 28952 leader_election.cc:302] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6987c4a92a844fe846d6c43319248bc; no voters:
I20260927 22:15:22.794713 28952 leader_election.cc:288] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:22.794838 28957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:22.795569 28957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [term 1 LEADER]: Becoming Leader. State: Replica: b6987c4a92a844fe846d6c43319248bc, State: Running, Role: LEADER
I20260927 22:15:22.796118 28957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } }
I20260927 22:15:22.796875 28952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:22.798741 28959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6987c4a92a844fe846d6c43319248bc. Latest consensus state: current_term: 1 leader_uuid: "b6987c4a92a844fe846d6c43319248bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } } }
I20260927 22:15:22.799393 28959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:22.799456 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6987c4a92a844fe846d6c43319248bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6987c4a92a844fe846d6c43319248bc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40703 } } }
I20260927 22:15:22.801414 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:22.801911 28965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:22.807497 28965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:22.815521 28965 catalog_manager.cc:1347] Generated new cluster ID: 325251632e7d4788aeadc0c10c508ebb
I20260927 22:15:22.815645 28965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:22.829566 28965 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:22.830009 28965 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:22.840914 28965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: Generated new TSK 0
I20260927 22:15:22.841770 28965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:15:22.854148 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--builtin_ntp_servers=127.23.173.84:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:15:23.175499 28976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:23.175755 28976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:23.175843 28976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:23.182909 28976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:23.183001 28976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:23.183115 28976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:23.191823 28976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:23.193013 28976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:23.194720 28976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:23.207858 28981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:23.207931 28982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:23.207901 28984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:23.208784 28976 server_base.cc:1034] running on GCE node
I20260927 22:15:23.209393 28976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:23.210397 28976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:23.211630 28976 hybrid_clock.cc:648] HybridClock initialized: now 1790547323211593 us; error 53 us; skew 500 ppm
I20260927 22:15:23.211947 28976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:23.214231 28976 webserver.cc:460] Webserver started at http://127.23.173.65:46125/ using document root <none> and password file <none>
I20260927 22:15:23.214937 28976 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:23.215075 28976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:23.215400 28976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:23.218266 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "456f2a5ad33640fdbaec6dcdb4463547"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "63e6164453229589f1fdfbbf2a8b4bb6"
server_key_iv: "0462d93557df95339b767c13bc0c77d1"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.219055 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "456f2a5ad33640fdbaec6dcdb4463547"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "63e6164453229589f1fdfbbf2a8b4bb6"
server_key_iv: "0462d93557df95339b767c13bc0c77d1"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.224035 28976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:15:23.227381 28990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.228678 28976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:23.228854 28976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "456f2a5ad33640fdbaec6dcdb4463547"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "63e6164453229589f1fdfbbf2a8b4bb6"
server_key_iv: "0462d93557df95339b767c13bc0c77d1"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.229068 28976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:23.234686 28976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:23.235445 28976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:23.235721 28976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:23.236642 28976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:23.238142 28976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:23.238261 28976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.238394 28976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:23.238482 28976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.270861 28976 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:44541
I20260927 22:15:23.270916 29102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:44541 every 8 connection(s)
I20260927 22:15:23.272207 28976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:23.273257 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 28976
I20260927 22:15:23.273432 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:23.274210 24245 external_mini_cluster.cc:1428] Setting key 49cc3c6e7908bfa3dbd7d19500a1619c
I20260927 22:15:23.277024 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--builtin_ntp_servers=127.23.173.84:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:15:23.281605 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:23.275328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:39879 (local address 127.23.173.126:40703)
0927 22:15:23.275534 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:23.275540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:23.277292 (+ 1752us) server_negotiation.cc:407] Connection header received
0927 22:15:23.279181 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:23.279187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:23.279253 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:23.279373 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:23.280781 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:23.280787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:23.280803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:23.280830 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:23.280870 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:23.280899 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:23.280977 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:23.281268 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:15:23.281337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:15:23.282626 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:23.275963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40703 (local address 127.23.173.65:39879)
0927 22:15:23.277145 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:23.277190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:23.278906 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:23.279611 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:23.279619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:23.279713 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:23.280436 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:23.280439 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:23.280477 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:23.280564 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:23.281051 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:23.281059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:23.281064 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:15:23.281345 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:15:23.281599 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":361,"threads_started":1}
I20260927 22:15:23.284271 29103 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40703
I20260927 22:15:23.284569 29103 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:23.285106 29103 heartbeater.cc:511] Master 127.23.173.126:40703 requested a full tablet report, sending...
I20260927 22:15:23.287055 28917 ts_manager.cc:194] Registered new tserver with Master: 456f2a5ad33640fdbaec6dcdb4463547 (127.23.173.65:44541)
W20260927 22:15:23.606678 29106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:23.606961 29106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:23.607165 29106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:23.614401 29106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:23.614497 29106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:23.614709 29106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:23.623682 29106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:23.625046 29106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:23.626854 29106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:23.639952 29114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:23.639981 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:23.640048 29111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:23.640578 29106 server_base.cc:1034] running on GCE node
I20260927 22:15:23.641273 29106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:23.642264 29106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:23.643501 29106 hybrid_clock.cc:648] HybridClock initialized: now 1790547323643454 us; error 53 us; skew 500 ppm
I20260927 22:15:23.643815 29106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:23.646116 29106 webserver.cc:460] Webserver started at http://127.23.173.66:46501/ using document root <none> and password file <none>
I20260927 22:15:23.646783 29106 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:23.646921 29106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:23.647264 29106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:23.650027 29106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "eeeccf1657f749d0b2742af5b3284b82"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "9aaccfb021ec37793cc20a50dd8c4a9f"
server_key_iv: "0b746698b5bf4803e010e7d2e4aeb69b"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.650826 29106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "eeeccf1657f749d0b2742af5b3284b82"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "9aaccfb021ec37793cc20a50dd8c4a9f"
server_key_iv: "0b746698b5bf4803e010e7d2e4aeb69b"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.655864 29106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:15:23.659162 29120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.660415 29106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:23.660591 29106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "eeeccf1657f749d0b2742af5b3284b82"
format_stamp: "Formatted at 2026-09-27 22:15:23 on dist-test-slave-3wc3"
server_key: "9aaccfb021ec37793cc20a50dd8c4a9f"
server_key_iv: "0b746698b5bf4803e010e7d2e4aeb69b"
server_key_version: "encryptionkey@0"
I20260927 22:15:23.660802 29106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:23.666400 29106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:23.667192 29106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:23.667464 29106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:23.668335 29106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:23.669668 29106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:23.669787 29106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.669909 29106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:23.670018 29106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:23.703337 29106 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:45999
I20260927 22:15:23.703369 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:45999 every 8 connection(s)
I20260927 22:15:23.704662 29106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:23.708961 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29106
I20260927 22:15:23.709141 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:23.709708 24245 external_mini_cluster.cc:1428] Setting key b086e59a0bc61d5316e8207af7a660b5
I20260927 22:15:23.712786 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--builtin_ntp_servers=127.23.173.84:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:15:23.713666 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:23.707277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50811 (local address 127.23.173.126:40703)
0927 22:15:23.707501 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:23.707508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:23.709164 (+ 1656us) server_negotiation.cc:407] Connection header received
0927 22:15:23.711275 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:23.711281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:23.711345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:23.711470 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:23.712965 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:23.712973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:23.712980 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:23.713005 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:23.713050 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:23.713076 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:23.713178 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:23.713405 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:15:23.713454 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:15:23.715001 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:23.707918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40703 (local address 127.23.173.66:50811)
0927 22:15:23.708931 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:23.708984 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:23.711013 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:23.711739 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:23.711750 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:23.711875 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:23.712577 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:23.712582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:23.712637 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:23.712749 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:23.713173 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:23.713186 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:23.713197 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:15:23.713478 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:15:23.713743 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
I20260927 22:15:23.716361 29233 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40703
I20260927 22:15:23.716665 29233 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:23.717342 29233 heartbeater.cc:511] Master 127.23.173.126:40703 requested a full tablet report, sending...
I20260927 22:15:23.718806 28917 ts_manager.cc:194] Registered new tserver with Master: eeeccf1657f749d0b2742af5b3284b82 (127.23.173.66:45999)
W20260927 22:15:24.035544 29236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:24.035851 29236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:24.036006 29236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:24.043152 29236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:24.043279 29236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:24.043460 29236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:24.053246 29236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:24.054921 29236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:24.057229 29236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:24.071808 29242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:24.072190 29241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:24.072393 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:24.073264 29236 server_base.cc:1034] running on GCE node
I20260927 22:15:24.073887 29236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:24.074798 29236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:24.076020 29236 hybrid_clock.cc:648] HybridClock initialized: now 1790547324075967 us; error 41 us; skew 500 ppm
I20260927 22:15:24.076337 29236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:24.078478 29236 webserver.cc:460] Webserver started at http://127.23.173.67:35111/ using document root <none> and password file <none>
I20260927 22:15:24.079135 29236 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:24.079246 29236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:24.079603 29236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:24.082371 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "a5b5a9720ffb4322a3d5a21716801f2f"
format_stamp: "Formatted at 2026-09-27 22:15:24 on dist-test-slave-3wc3"
server_key: "6af462d7167db4d986e29e29ae8d650d"
server_key_iv: "04107d065209ab80fb38910a20cf75ea"
server_key_version: "encryptionkey@0"
I20260927 22:15:24.083133 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b5a9720ffb4322a3d5a21716801f2f"
format_stamp: "Formatted at 2026-09-27 22:15:24 on dist-test-slave-3wc3"
server_key: "6af462d7167db4d986e29e29ae8d650d"
server_key_iv: "04107d065209ab80fb38910a20cf75ea"
server_key_version: "encryptionkey@0"
I20260927 22:15:24.087849 29236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:15:24.090986 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:24.092187 29236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:15:24.092378 29236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "a5b5a9720ffb4322a3d5a21716801f2f"
format_stamp: "Formatted at 2026-09-27 22:15:24 on dist-test-slave-3wc3"
server_key: "6af462d7167db4d986e29e29ae8d650d"
server_key_iv: "04107d065209ab80fb38910a20cf75ea"
server_key_version: "encryptionkey@0"
I20260927 22:15:24.092625 29236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:24.098129 29236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:24.098865 29236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:24.099125 29236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:24.100029 29236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:24.101435 29236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:24.101563 29236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:24.101681 29236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:24.101768 29236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:24.135732 29236 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:37421
I20260927 22:15:24.135839 29362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:37421 every 8 connection(s)
I20260927 22:15:24.137185 29236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:24.144994 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29236
I20260927 22:15:24.145159 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:24.145190 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.139868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:37783 (local address 127.23.173.126:40703)
0927 22:15:24.140101 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.140108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.141438 (+ 1330us) server_negotiation.cc:407] Connection header received
0927 22:15:24.143123 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.143127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.143180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.143265 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.144465 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.144470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.144473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.144488 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.144519 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.144530 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.144621 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.144877 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.144928 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:15:24.145782 24245 external_mini_cluster.cc:1428] Setting key 40de48fd3c579ef3acc8b40384a74f27
I20260927 22:15:24.146471 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.140400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40703 (local address 127.23.173.67:37783)
0927 22:15:24.141288 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.141331 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.142882 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.143464 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.143472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.143563 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.144192 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.144194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.144229 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.144310 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.144629 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.144640 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.144648 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.144954 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.145235 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":265,"threads_started":1}
I20260927 22:15:24.147818 29363 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40703
I20260927 22:15:24.148108 29363 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:24.148682 29363 heartbeater.cc:511] Master 127.23.173.126:40703 requested a full tablet report, sending...
I20260927 22:15:24.150063 28917 ts_manager.cc:194] Registered new tserver with Master: a5b5a9720ffb4322a3d5a21716801f2f (127.23.173.67:37421)
I20260927 22:15:24.161054 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:24.167879 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.165546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.23.173.126:40703)
0927 22:15:24.165731 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.165736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.166311 (+ 575us) server_negotiation.cc:407] Connection header received
0927 22:15:24.166437 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.166440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.166491 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.166573 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.167377 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.167381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.167385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.167400 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.167432 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.167444 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.167537 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.167622 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.167667 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:15:24.171530 28917 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:24.178406 28917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:24.208745 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.190502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.23.173.66:45999)
0927 22:15:24.203310 (+ 12808us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.203322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.204426 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 22:15:24.204511 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.204522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.204794 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.204950 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.207577 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.207590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.207607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.207638 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.207700 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.207722 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.207857 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.208198 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.208310 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12553,"thread_start_us":12823,"threads_started":1}
I20260927 22:15:24.209753 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.189817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.23.173.65:44541)
0927 22:15:24.207217 (+ 17400us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.207226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.207257 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:15:24.207322 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.207329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.207544 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.207678 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.209008 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.209016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.209027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.209054 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.209093 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.209107 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.209255 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.209332 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.209421 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16863,"thread_start_us":16743,"threads_started":1}
I20260927 22:15:24.210083 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.189805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58202)
0927 22:15:24.205366 (+ 15561us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.205387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.205497 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.207757 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.207762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.207784 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.208581 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.208584 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.208600 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.208644 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.209215 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.209220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.209225 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.209362 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.209415 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15410,"thread_start_us":15252,"threads_started":1}
I20260927 22:15:24.210243 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.189783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:45999 (local address 127.0.0.1:58552)
0927 22:15:24.204097 (+ 14314us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.204137 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.204288 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.205999 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.206011 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.206136 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.207275 (+ 1139us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.207280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.207311 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.207390 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.207964 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.207976 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.207983 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.209152 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.209252 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14048,"thread_start_us":13967,"threads_started":1}
I20260927 22:15:24.210559 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.204951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.23.173.67:37421)
0927 22:15:24.205789 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.205800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.207300 (+ 1500us) server_negotiation.cc:407] Connection header received
0927 22:15:24.207386 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.207394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.207613 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.207762 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.209041 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.209049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.209062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.209092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.209149 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.209171 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.209325 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.210048 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.210158 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":444,"threads_started":1}
I20260927 22:15:24.210767 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.205741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37421 (local address 127.0.0.1:47120)
0927 22:15:24.207061 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.207077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.207173 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.208176 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.208183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.208206 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.208831 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.208837 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.208849 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.208887 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.209379 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.209385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.209396 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.209523 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.209573 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":793,"threads_started":1}
I20260927 22:15:24.214300 29038 tablet_service.cc:1467] Processing CreateTablet for tablet b7aec5de6917413b957267de0eaed6fa (DEFAULT_TABLE table=test-table [id=0102ae4d4c0a496c8902a6b34d90aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:24.214679 29168 tablet_service.cc:1467] Processing CreateTablet for tablet b7aec5de6917413b957267de0eaed6fa (DEFAULT_TABLE table=test-table [id=0102ae4d4c0a496c8902a6b34d90aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:24.215801 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7aec5de6917413b957267de0eaed6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:24.215971 29168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7aec5de6917413b957267de0eaed6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:24.216316 29298 tablet_service.cc:1467] Processing CreateTablet for tablet b7aec5de6917413b957267de0eaed6fa (DEFAULT_TABLE table=test-table [id=0102ae4d4c0a496c8902a6b34d90aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:24.217355 29298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7aec5de6917413b957267de0eaed6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:24.228022 29381 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Bootstrap starting.
I20260927 22:15:24.231262 29383 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Bootstrap starting.
I20260927 22:15:24.231652 29382 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Bootstrap starting.
I20260927 22:15:24.232697 29381 tablet_bootstrap.cc:654] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:24.234107 29381 log.cc:824] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:24.235360 29383 tablet_bootstrap.cc:654] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:24.235633 29382 tablet_bootstrap.cc:654] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:24.236821 29382 log.cc:824] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:24.237900 29383 log.cc:824] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:24.239532 29382 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: No bootstrap required, opened a new log
I20260927 22:15:24.239815 29382 ts_tablet_manager.cc:1397] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260927 22:15:24.240458 29383 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: No bootstrap required, opened a new log
I20260927 22:15:24.240707 29383 ts_tablet_manager.cc:1397] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:15:24.243798 29381 tablet_bootstrap.cc:492] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: No bootstrap required, opened a new log
I20260927 22:15:24.244066 29381 ts_tablet_manager.cc:1397] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260927 22:15:24.247862 29382 raft_consensus.cc:348] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.248183 29382 raft_consensus.cc:374] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:24.248246 29382 raft_consensus.cc:729] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeeccf1657f749d0b2742af5b3284b82, State: Initialized, Role: FOLLOWER
I20260927 22:15:24.248826 29382 consensus_queue.cc:260] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.249876 29233 heartbeater.cc:503] Master 127.23.173.126:40703 was elected leader, sending a full tablet report...
I20260927 22:15:24.250578 29382 ts_tablet_manager.cc:1428] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260927 22:15:24.251006 29383 raft_consensus.cc:348] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.251401 29383 raft_consensus.cc:374] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:24.251550 29383 raft_consensus.cc:729] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456f2a5ad33640fdbaec6dcdb4463547, State: Initialized, Role: FOLLOWER
I20260927 22:15:24.252095 29383 consensus_queue.cc:260] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.253154 29103 heartbeater.cc:503] Master 127.23.173.126:40703 was elected leader, sending a full tablet report...
I20260927 22:15:24.253768 29383 ts_tablet_manager.cc:1428] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:15:24.255115 29381 raft_consensus.cc:348] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.255448 29381 raft_consensus.cc:374] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:24.255596 29381 raft_consensus.cc:729] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b5a9720ffb4322a3d5a21716801f2f, State: Initialized, Role: FOLLOWER
I20260927 22:15:24.256204 29381 consensus_queue.cc:260] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.257373 29363 heartbeater.cc:503] Master 127.23.173.126:40703 was elected leader, sending a full tablet report...
I20260927 22:15:24.257901 29381 ts_tablet_manager.cc:1428] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260927 22:15:24.274718 29104 tablet.cc:2369] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:24.389002 29364 tablet.cc:2369] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:24.456883 29234 tablet.cc:2369] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:15:24.466902 29389 raft_consensus.cc:482] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:24.467221 29389 raft_consensus.cc:504] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.468923 29389 leader_election.cc:288] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 456f2a5ad33640fdbaec6dcdb4463547 (127.23.173.65:44541), eeeccf1657f749d0b2742af5b3284b82 (127.23.173.66:45999)
I20260927 22:15:24.471776 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.469364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.23.173.67:58707)
0927 22:15:24.469715 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.469743 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.469886 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.470246 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.470252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.470275 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.470935 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.470937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.470951 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.470991 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.471319 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.471323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.471328 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.471465 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.471520 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260927 22:15:24.471774 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.469402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:58707 (local address 127.23.173.65:44541)
0927 22:15:24.469605 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.469613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.469829 (+ 216us) server_negotiation.cc:407] Connection header received
0927 22:15:24.470031 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.470047 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.470100 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.470209 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.471140 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.471145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.471150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.471170 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.471204 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.471215 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.471333 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.471464 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.471523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:15:24.472257 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.469454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:60527 (local address 127.23.173.66:45999)
0927 22:15:24.469649 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.469655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.470376 (+ 721us) server_negotiation.cc:407] Connection header received
0927 22:15:24.470557 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.470562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.470616 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.470722 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.471663 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.471669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.471673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.471698 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.471727 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.471737 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.471869 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.471947 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.472006 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:15:24.472424 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.469470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:45999 (local address 127.23.173.67:60527)
0927 22:15:24.470245 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.470259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:24.470409 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:24.470737 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:24.470742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:24.470765 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:24.471457 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:15:24.471459 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:24.471472 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:15:24.471506 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:24.471832 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:24.471837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:24.471843 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:15:24.471951 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:15:24.472001 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":441,"threads_started":1}
I20260927 22:15:24.473120 29058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7aec5de6917413b957267de0eaed6fa" candidate_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456f2a5ad33640fdbaec6dcdb4463547" is_pre_election: true
I20260927 22:15:24.473385 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7aec5de6917413b957267de0eaed6fa" candidate_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeeccf1657f749d0b2742af5b3284b82" is_pre_election: true
I20260927 22:15:24.473747 29058 raft_consensus.cc:2454] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b5a9720ffb4322a3d5a21716801f2f in term 0.
I20260927 22:15:24.473907 29188 raft_consensus.cc:2454] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b5a9720ffb4322a3d5a21716801f2f in term 0.
I20260927 22:15:24.474483 29252 leader_election.cc:302] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456f2a5ad33640fdbaec6dcdb4463547, a5b5a9720ffb4322a3d5a21716801f2f; no voters:
I20260927 22:15:24.474876 29389 raft_consensus.cc:2781] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:24.475021 29389 raft_consensus.cc:482] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:24.475127 29389 raft_consensus.cc:3037] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:24.477373 29389 raft_consensus.cc:504] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.478170 29389 leader_election.cc:288] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [CANDIDATE]: Term 1 election: Requested vote from peers 456f2a5ad33640fdbaec6dcdb4463547 (127.23.173.65:44541), eeeccf1657f749d0b2742af5b3284b82 (127.23.173.66:45999)
I20260927 22:15:24.478540 29058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7aec5de6917413b957267de0eaed6fa" candidate_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456f2a5ad33640fdbaec6dcdb4463547"
I20260927 22:15:24.478657 29188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7aec5de6917413b957267de0eaed6fa" candidate_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeeccf1657f749d0b2742af5b3284b82"
I20260927 22:15:24.478757 29058 raft_consensus.cc:3037] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:24.478893 29188 raft_consensus.cc:3037] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:24.480935 29188 raft_consensus.cc:2454] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b5a9720ffb4322a3d5a21716801f2f in term 1.
I20260927 22:15:24.480952 29058 raft_consensus.cc:2454] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b5a9720ffb4322a3d5a21716801f2f in term 1.
I20260927 22:15:24.481359 29254 leader_election.cc:302] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b5a9720ffb4322a3d5a21716801f2f, eeeccf1657f749d0b2742af5b3284b82; no voters:
I20260927 22:15:24.481659 29389 raft_consensus.cc:2781] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:24.482163 29389 raft_consensus.cc:686] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 1 LEADER]: Becoming Leader. State: Replica: a5b5a9720ffb4322a3d5a21716801f2f, State: Running, Role: LEADER
I20260927 22:15:24.482635 29389 consensus_queue.cc:237] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } }
I20260927 22:15:24.486992 28916 catalog_manager.cc:5495] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b5a9720ffb4322a3d5a21716801f2f (127.23.173.67). New cstate: current_term: 1 leader_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b5a9720ffb4322a3d5a21716801f2f" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37421 } health_report { overall_health: HEALTHY } } }
I20260927 22:15:24.507825 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.505664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.23.173.67:37421)
0927 22:15:24.505849 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.505855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.505972 (+ 117us) server_negotiation.cc:407] Connection header received
0927 22:15:24.506235 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.506239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.506313 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.506420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.507232 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.507238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.507242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.507260 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.507298 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.507309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.507437 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.507527 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.507585 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:15:24.515738 29188 raft_consensus.cc:1261] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 1 FOLLOWER]: Refusing update from remote peer a5b5a9720ffb4322a3d5a21716801f2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:24.515838 29058 raft_consensus.cc:1261] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 1 FOLLOWER]: Refusing update from remote peer a5b5a9720ffb4322a3d5a21716801f2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:24.516746 29391 consensus_queue.cc:1035] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeeccf1657f749d0b2742af5b3284b82" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:24.517057 29389 consensus_queue.cc:1035] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "456f2a5ad33640fdbaec6dcdb4463547" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:24.529302 29394 mvcc.cc:205] Tried to move back new op lower bound from 7334081841207078912 to 7334081841084411904. Current Snapshot: MvccSnapshot[applied={T|T < 7334081841207078912}]
I20260927 22:15:24.544109 28917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:24.544325 28917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:24.547816 28917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6987c4a92a844fe846d6c43319248bc: Sending DeleteTablet for 3 replicas of tablet b7aec5de6917413b957267de0eaed6fa
I20260927 22:15:24.549011 29038 tablet_service.cc:1514] Processing DeleteTablet for tablet b7aec5de6917413b957267de0eaed6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:24 UTC) from {username='slave'} at 127.0.0.1:58202
I20260927 22:15:24.549104 29168 tablet_service.cc:1514] Processing DeleteTablet for tablet b7aec5de6917413b957267de0eaed6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:24 UTC) from {username='slave'} at 127.0.0.1:58552
I20260927 22:15:24.549196 29298 tablet_service.cc:1514] Processing DeleteTablet for tablet b7aec5de6917413b957267de0eaed6fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:24 UTC) from {username='slave'} at 127.0.0.1:47120
I20260927 22:15:24.550676 29406 tablet_replica.cc:330] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: stopping tablet replica
I20260927 22:15:24.551180 29406 raft_consensus.cc:2229] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:24.551512 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28976
I20260927 22:15:24.551923 29406 raft_consensus.cc:2258] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:24.552906 29405 tablet_replica.cc:330] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: stopping tablet replica
I20260927 22:15:24.553287 29405 raft_consensus.cc:2229] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:24.553812 29405 raft_consensus.cc:2258] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:24.554268 29406 ts_tablet_manager.cc:1905] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:24.555346 29404 tablet_replica.cc:330] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547: stopping tablet replica
I20260927 22:15:24.555709 29404 raft_consensus.cc:2229] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:24.556154 29404 raft_consensus.cc:2258] T b7aec5de6917413b957267de0eaed6fa P 456f2a5ad33640fdbaec6dcdb4463547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:24.558230 29405 ts_tablet_manager.cc:1905] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:24.558631 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.554048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.23.173.65:44541)
0927 22:15:24.554283 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.554292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.555762 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 22:15:24.555822 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.555825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.555909 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.556026 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.557033 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.557051 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.557056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.557073 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.557111 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.557126 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.557238 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.558292 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.558352 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:15:24.559474 29098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:24.565425 29406 ts_tablet_manager.cc:1918] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:24.565678 29406 log.cc:1196] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/b7aec5de6917413b957267de0eaed6fa
I20260927 22:15:24.566354 29406 ts_tablet_manager.cc:1939] T b7aec5de6917413b957267de0eaed6fa P a5b5a9720ffb4322a3d5a21716801f2f: Deleting consensus metadata
I20260927 22:15:24.566570 29405 ts_tablet_manager.cc:1918] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:24.566772 29405 log.cc:1196] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/b7aec5de6917413b957267de0eaed6fa
I20260927 22:15:24.567291 29405 ts_tablet_manager.cc:1939] T b7aec5de6917413b957267de0eaed6fa P eeeccf1657f749d0b2742af5b3284b82: Deleting consensus metadata
I20260927 22:15:24.568477 28904 catalog_manager.cc:4841] TS a5b5a9720ffb4322a3d5a21716801f2f (127.23.173.67:37421): tablet b7aec5de6917413b957267de0eaed6fa (table test-table [id=0102ae4d4c0a496c8902a6b34d90aeff]) successfully deleted
I20260927 22:15:24.569001 28904 catalog_manager.cc:4841] TS eeeccf1657f749d0b2742af5b3284b82 (127.23.173.66:45999): tablet b7aec5de6917413b957267de0eaed6fa (table test-table [id=0102ae4d4c0a496c8902a6b34d90aeff]) successfully deleted
I20260927 22:15:24.675266 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28976
W20260927 22:15:24.693248 28902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.65:44541 (error 108)
I20260927 22:15:24.693339 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29106
I20260927 22:15:24.695149 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.694678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58218)
0927 22:15:24.694845 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.694968 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:15:24.695408 28902 catalog_manager.cc:4584] TS 456f2a5ad33640fdbaec6dcdb4463547 (127.23.173.65:44541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7aec5de6917413b957267de0eaed6fa: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
I20260927 22:15:24.696728 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.694361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.23.173.66:45999)
0927 22:15:24.694571 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.694578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.694599 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:15:24.694839 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.694847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.694958 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.695062 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.696053 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.696060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.696065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.696084 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.696124 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.696140 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.696285 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.696377 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.696435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:15:24.697476 29228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:24.749156 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.748694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58228)
0927 22:15:24.748911 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.748999 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:15:24.750226 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.749855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58240)
0927 22:15:24.750033 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.750120 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:15:24.810657 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.810196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58246)
0927 22:15:24.810389 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.810503 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:15:24.811724 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.811372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58248)
0927 22:15:24.811520 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.811616 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:15:24.847206 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29106
I20260927 22:15:24.864049 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29236
I20260927 22:15:24.867100 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.864879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.23.173.67:37421)
0927 22:15:24.865086 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:24.865093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:24.865115 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:15:24.865256 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:24.865264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:24.865337 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:24.865455 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:24.866382 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:24.866388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:24.866393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:24.866417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:24.866457 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:24.866473 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:24.866625 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:24.866682 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 22:15:24.866752 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:15:24.867905 29358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:24.892532 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.891944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58260)
0927 22:15:24.892211 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.892329 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:15:24.893620 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:24.893185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58274)
0927 22:15:24.893399 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:24.893480 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 22:15:24.988057 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29236
I20260927 22:15:25.007398 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 28886
I20260927 22:15:25.008585 28947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:25.114773 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:25.114041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:44541 (local address 127.0.0.1:58278)
0927 22:15:25.114359 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:25.114539 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:44541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:15:25.114863 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 28886
2026-09-27T22:15:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2818 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:15:25.155452 24245 mini_oidc.cc:152] Starting JWKS server
I20260927 22:15:25.157920 24245 webserver.cc:460] Webserver started at https://127.0.0.1:37253/ using document root <none> and password file <none>
I20260927 22:15:25.158267 24245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:15:25.158901 24245 webserver.cc:460] Webserver started at http://127.0.0.1:37955/ using document root <none> and password file <none>
2026-09-27T22:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:25Z Disabled control of system clock
I20260927 22:15:25.186566 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40459
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:40459
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/testchainca.cert with env {}
W20260927 22:15:25.502938 29418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:25.503218 29418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:25.503305 29418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:25.510298 29418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:25.510411 29418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:25.510448 29418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:25.510504 29418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:25.510548 29418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:15:25.510620 29418 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:15:25.510660 29418 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37253/jwks.json
I20260927 22:15:25.520365 29418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:40459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:25.521559 29418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:25.522841 29418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:25.534335 29423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:25.534260 29426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:25.534399 29424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:25.535554 29418 server_base.cc:1034] running on GCE node
I20260927 22:15:25.536285 29418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:25.537596 29418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:25.538825 29418 hybrid_clock.cc:648] HybridClock initialized: now 1790547325538779 us; error 36 us; skew 500 ppm
I20260927 22:15:25.539173 29418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:25.545761 29418 webserver.cc:460] Webserver started at http://127.23.173.126:33759/ using document root <none> and password file <none>
I20260927 22:15:25.546509 29418 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:25.546660 29418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:25.546976 29418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:25.549803 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "7515fc063d1c4930ba263353da73dce3"
format_stamp: "Formatted at 2026-09-27 22:15:25 on dist-test-slave-3wc3"
server_key: "44587560e8a53389ccace62bc501df7f"
server_key_iv: "ac56bb8189a1a1e7e50d5399431eee89"
server_key_version: "encryptionkey@0"
I20260927 22:15:25.550674 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "7515fc063d1c4930ba263353da73dce3"
format_stamp: "Formatted at 2026-09-27 22:15:25 on dist-test-slave-3wc3"
server_key: "44587560e8a53389ccace62bc501df7f"
server_key_iv: "ac56bb8189a1a1e7e50d5399431eee89"
server_key_version: "encryptionkey@0"
I20260927 22:15:25.555697 29418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:15:25.559136 29432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:25.560860 29418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:25.561045 29418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "7515fc063d1c4930ba263353da73dce3"
format_stamp: "Formatted at 2026-09-27 22:15:25 on dist-test-slave-3wc3"
server_key: "44587560e8a53389ccace62bc501df7f"
server_key_iv: "ac56bb8189a1a1e7e50d5399431eee89"
server_key_version: "encryptionkey@0"
I20260927 22:15:25.561324 29418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:25.586897 29418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:25.587764 29418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:25.588083 29418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:25.606287 29418 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:40459
I20260927 22:15:25.606308 29483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:40459 every 8 connection(s)
I20260927 22:15:25.607637 29418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:25.611192 29484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:25.614094 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29418
I20260927 22:15:25.614310 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:25.615078 24245 external_mini_cluster.cc:1428] Setting key 6e725f4ac28f19a3e686cc01ef2bf555
I20260927 22:15:25.623754 29484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3: Bootstrap starting.
I20260927 22:15:25.627816 29484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:25.629057 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:25.617000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.23.173.126:40459)
0927 22:15:25.618062 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:25.618074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:25.618129 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:15:25.619764 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:25.619785 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:25.620235 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:25.620624 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:25.621198 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.622576 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.623937 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.624909 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.625122 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:25.626257 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:25.626266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:25.626283 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:25.626971 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:25.627099 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:25.627126 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:25.627236 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:25.627408 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:15:25.627739 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":314,"threads_started":1}
I20260927 22:15:25.629099 29484 log.cc:824] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:25.632035 29484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3: No bootstrap required, opened a new log
I20260927 22:15:25.632812 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:15:25.639354 29484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } }
I20260927 22:15:25.639668 29484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:25.639757 29484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7515fc063d1c4930ba263353da73dce3, State: Initialized, Role: FOLLOWER
I20260927 22:15:25.640301 29484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } }
I20260927 22:15:25.640520 29484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:25.640677 29484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:25.640848 29484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:25.643206 29484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } }
I20260927 22:15:25.643712 29484 leader_election.cc:302] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7515fc063d1c4930ba263353da73dce3; no voters:
I20260927 22:15:25.644258 29484 leader_election.cc:288] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:25.644416 29489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:25.645113 29489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [term 1 LEADER]: Becoming Leader. State: Replica: 7515fc063d1c4930ba263353da73dce3, State: Running, Role: LEADER
I20260927 22:15:25.645633 29489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } }
I20260927 22:15:25.646365 29484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:25.648805 29491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7515fc063d1c4930ba263353da73dce3. Latest consensus state: current_term: 1 leader_uuid: "7515fc063d1c4930ba263353da73dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } } }
I20260927 22:15:25.649250 29491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:25.649583 29490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7515fc063d1c4930ba263353da73dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7515fc063d1c4930ba263353da73dce3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40459 } } }
I20260927 22:15:25.649972 29490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:25.651474 29498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:25.655702 29498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:25.664420 29498 catalog_manager.cc:1347] Generated new cluster ID: f6792e4d479545b581b2e64d111f9547
I20260927 22:15:25.664530 29498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:25.706962 29498 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:25.708328 29498 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:25.715987 29498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7515fc063d1c4930ba263353da73dce3: Generated new TSK 0
I20260927 22:15:25.716848 29498 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:15:25.769244 29514 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:15:25.769580 29514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:15:25.771291 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:25.740937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.23.173.126:40459)
0927 22:15:25.741185 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:25.741195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:25.741622 (+ 427us) server_negotiation.cc:407] Connection header received
0927 22:15:25.742030 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:25.742035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:25.742115 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:25.742237 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:15:25.742894 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.743601 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.745701 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.746215 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.746303 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:25.746948 (+ 645us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:15:25.769836 (+ 22888us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:15:25.770933 (+ 1097us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:15:25.771422 29487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:15:25.774372 29513 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:25.740845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40459 (local address 127.0.0.1:45062)
0927 22:15:25.741508 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:25.741532 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:25.741724 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:25.742298 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:25.742304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:25.742317 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:15:25.742720 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:25.742734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.743744 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.743750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:25.745558 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:25.745573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:25.746388 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:25.746397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:25.746590 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:25.746663 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:15:25.770259 (+ 23596us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:15:25.772253 (+ 1994us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.126:40459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":310,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.173.126:40459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:15:25.775875 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29418
I20260927 22:15:25.777066 29479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:25.874704 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29418
2026-09-27T22:15:25Z chronyd exiting
I20260927 22:15:25.924576 24245 test_util.cc:175] -----------------------------------------------
I20260927 22:15:25.924711 24245 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790547283249637-24245-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:15:25.928359 24245 mini_oidc.cc:152] Starting JWKS server
I20260927 22:15:25.930202 24245 webserver.cc:460] Webserver started at https://127.0.0.1:45211/ using document root <none> and password file <none>
I20260927 22:15:25.930428 24245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:15:25.931128 24245 webserver.cc:460] Webserver started at http://127.0.0.1:42411/ using document root <none> and password file <none>
2026-09-27T22:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:25Z Disabled control of system clock
I20260927 22:15:25.956357 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:44867
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:44867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45211/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/testchainca.cert with env {}
W20260927 22:15:26.275857 29521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:26.276201 29521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:26.276360 29521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:26.283588 29521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:26.283725 29521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:26.283816 29521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:26.283936 29521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:26.284013 29521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:15:26.284108 29521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:15:26.284188 29521 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45211/jwks.json
I20260927 22:15:26.293887 29521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:44867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45211/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:26.295173 29521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:26.296473 29521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:26.307869 29527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:26.307891 29529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:26.308269 29526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:26.308671 29521 server_base.cc:1034] running on GCE node
I20260927 22:15:26.309437 29521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:26.310914 29521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:26.312147 29521 hybrid_clock.cc:648] HybridClock initialized: now 1790547326312101 us; error 50 us; skew 500 ppm
I20260927 22:15:26.312494 29521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:26.314800 29521 webserver.cc:460] Webserver started at http://127.23.173.126:43129/ using document root <none> and password file <none>
I20260927 22:15:26.315487 29521 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:26.315629 29521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:26.315966 29521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:26.318919 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "df32216cd98e4903a6dea8352e4a26ed"
format_stamp: "Formatted at 2026-09-27 22:15:26 on dist-test-slave-3wc3"
server_key: "467fc745af0780a538479810febc5577"
server_key_iv: "2879105569a32ff9d64648e9ce1b0179"
server_key_version: "encryptionkey@0"
I20260927 22:15:26.319710 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "df32216cd98e4903a6dea8352e4a26ed"
format_stamp: "Formatted at 2026-09-27 22:15:26 on dist-test-slave-3wc3"
server_key: "467fc745af0780a538479810febc5577"
server_key_iv: "2879105569a32ff9d64648e9ce1b0179"
server_key_version: "encryptionkey@0"
I20260927 22:15:26.324663 29521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:15:26.327942 29535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:26.329221 29521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:26.329409 29521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "df32216cd98e4903a6dea8352e4a26ed"
format_stamp: "Formatted at 2026-09-27 22:15:26 on dist-test-slave-3wc3"
server_key: "467fc745af0780a538479810febc5577"
server_key_iv: "2879105569a32ff9d64648e9ce1b0179"
server_key_version: "encryptionkey@0"
I20260927 22:15:26.329638 29521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:26.350572 29521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:26.351444 29521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:26.351789 29521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:26.369772 29521 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:44867
I20260927 22:15:26.369792 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:44867 every 8 connection(s)
I20260927 22:15:26.371176 29521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:26.374431 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29521
I20260927 22:15:26.374634 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:26.374954 29587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:26.375321 24245 external_mini_cluster.cc:1428] Setting key 6c55ed6f852daa8f126db23ad4967f5d
I20260927 22:15:26.390460 29587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed: Bootstrap starting.
I20260927 22:15:26.390620 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:26.377241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.23.173.126:44867)
0927 22:15:26.378382 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:26.378396 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:26.378456 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:15:26.380213 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:26.380245 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:26.380873 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:26.381329 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:26.381884 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.383944 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.385334 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.386445 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.386643 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:26.387827 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:26.387838 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:26.387856 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:26.388579 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:26.388670 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:26.388694 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:26.388799 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:26.389009 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:15:26.389324 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":375,"threads_started":1}
I20260927 22:15:26.393774 29587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:26.394402 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:15:26.394922 29587 log.cc:824] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:26.397805 29587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed: No bootstrap required, opened a new log
I20260927 22:15:26.405282 29587 raft_consensus.cc:348] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } }
I20260927 22:15:26.405640 29587 raft_consensus.cc:374] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:26.405751 29587 raft_consensus.cc:729] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df32216cd98e4903a6dea8352e4a26ed, State: Initialized, Role: FOLLOWER
I20260927 22:15:26.406415 29587 consensus_queue.cc:260] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } }
I20260927 22:15:26.406688 29587 raft_consensus.cc:388] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:26.406842 29587 raft_consensus.cc:482] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:26.407017 29587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:26.409106 29587 raft_consensus.cc:504] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } }
I20260927 22:15:26.409593 29587 leader_election.cc:302] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df32216cd98e4903a6dea8352e4a26ed; no voters:
I20260927 22:15:26.410256 29587 leader_election.cc:288] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:26.410450 29592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:26.411366 29592 raft_consensus.cc:686] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [term 1 LEADER]: Becoming Leader. State: Replica: df32216cd98e4903a6dea8352e4a26ed, State: Running, Role: LEADER
I20260927 22:15:26.411907 29592 consensus_queue.cc:237] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } }
I20260927 22:15:26.412434 29587 sys_catalog.cc:564] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:26.414645 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df32216cd98e4903a6dea8352e4a26ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } } }
I20260927 22:15:26.414954 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:26.415395 29594 sys_catalog.cc:455] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader df32216cd98e4903a6dea8352e4a26ed. Latest consensus state: current_term: 1 leader_uuid: "df32216cd98e4903a6dea8352e4a26ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df32216cd98e4903a6dea8352e4a26ed" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44867 } } }
I20260927 22:15:26.415716 29594 sys_catalog.cc:458] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:26.415778 29598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:26.420986 29598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:26.430001 29598 catalog_manager.cc:1347] Generated new cluster ID: a02a5f80dda844539a4e51b91bcaf961
I20260927 22:15:26.430143 29598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:26.447999 29598 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:26.449757 29598 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:26.457129 29598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df32216cd98e4903a6dea8352e4a26ed: Generated new TSK 0
I20260927 22:15:26.458001 29598 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:15:26.488628 29617 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:15:26.488765 29617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:15:26.490028 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:26.470285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.23.173.126:44867)
0927 22:15:26.470482 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:26.470488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:26.470899 (+ 411us) server_negotiation.cc:407] Connection header received
0927 22:15:26.471100 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:26.471105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:26.471204 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:26.471376 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:15:26.471774 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.472887 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.474263 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.474961 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.475103 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:26.475542 (+ 439us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:15:26.489299 (+ 13757us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:15:26.489675 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49578: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:15:26.490083 29616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:26.470149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:44867 (local address 127.0.0.1:49578)
0927 22:15:26.470798 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:26.470814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:26.470932 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:26.471386 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:26.471389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:26.471398 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:15:26.471633 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:26.471641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.473020 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.473024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:26.474132 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:26.474143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:26.475111 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:26.475115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:26.475239 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:26.475270 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:15:26.489555 (+ 14285us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:15:26.489738 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.126:44867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":364,"threads_started":1}
W20260927 22:15:26.490183 29590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49578: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 22:15:26.490778 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29521
I20260927 22:15:26.491966 29582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:26.584120 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29521
2026-09-27T22:15:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:15:26.639182 24245 mini_oidc.cc:152] Starting JWKS server
I20260927 22:15:26.641086 24245 webserver.cc:460] Webserver started at https://127.0.0.1:33407/ using document root <none> and password file <none>
I20260927 22:15:26.641345 24245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:15:26.642074 24245 webserver.cc:460] Webserver started at http://127.0.0.1:33247/ using document root <none> and password file <none>
2026-09-27T22:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:26Z Disabled control of system clock
I20260927 22:15:26.669299 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:36597
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:36597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:15:26.989059 29624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:26.989353 29624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:26.989437 29624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:26.996632 29624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:26.996738 29624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:26.996778 29624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:26.996834 29624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:26.996878 29624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:15:26.996933 29624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:15:26.996973 29624 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33407/jwks.json
I20260927 22:15:27.006223 29624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:36597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:36597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:27.007503 29624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:27.009272 29624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:27.021193 29630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:27.021121 29629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:27.021222 29632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:27.021982 29624 server_base.cc:1034] running on GCE node
I20260927 22:15:27.022811 29624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:27.024201 29624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:27.025429 29624 hybrid_clock.cc:648] HybridClock initialized: now 1790547327025395 us; error 48 us; skew 500 ppm
I20260927 22:15:27.025817 29624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:27.028049 29624 webserver.cc:460] Webserver started at http://127.23.173.126:42861/ using document root <none> and password file <none>
I20260927 22:15:27.028749 29624 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:27.028887 29624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:27.029204 29624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:27.032100 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "b9ad41d3cacc41819523b9511047f47e"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "45cdb368d71911abc0df402e1a7d40d1"
server_key_iv: "a3eb7b14708c71c4bff3ee6c8fa7e546"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.032919 29624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "b9ad41d3cacc41819523b9511047f47e"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "45cdb368d71911abc0df402e1a7d40d1"
server_key_iv: "a3eb7b14708c71c4bff3ee6c8fa7e546"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.038035 29624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:15:27.041400 29638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:27.042809 29624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:15:27.042991 29624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "b9ad41d3cacc41819523b9511047f47e"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "45cdb368d71911abc0df402e1a7d40d1"
server_key_iv: "a3eb7b14708c71c4bff3ee6c8fa7e546"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.043210 29624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:27.070865 29624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:27.071722 29624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:27.072036 29624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:27.089722 29624 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:36597
I20260927 22:15:27.089743 29689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:36597 every 8 connection(s)
I20260927 22:15:27.091041 29624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:27.094667 29690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:27.098146 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29624
I20260927 22:15:27.098361 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:27.099146 24245 external_mini_cluster.cc:1428] Setting key 6fe79942fd333b81eaf56a0430576afb
I20260927 22:15:27.107555 29690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e: Bootstrap starting.
I20260927 22:15:27.111166 29690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:27.112269 29690 log.cc:824] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:27.112473 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:27.101017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.23.173.126:36597)
0927 22:15:27.102045 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:27.102056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:27.102099 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:15:27.103687 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:27.103709 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:27.104186 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:27.104522 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:27.105088 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.106424 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.108208 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.108864 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.109005 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.110237 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:27.110244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:27.110257 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:27.110778 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:27.110867 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:27.110887 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:27.110982 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:27.111220 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:15:27.111513 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260927 22:15:27.115557 29690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e: No bootstrap required, opened a new log
I20260927 22:15:27.116376 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:15:27.123186 29690 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } }
I20260927 22:15:27.123574 29690 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:27.123687 29690 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ad41d3cacc41819523b9511047f47e, State: Initialized, Role: FOLLOWER
I20260927 22:15:27.124284 29690 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } }
I20260927 22:15:27.124478 29690 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:27.124578 29690 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:27.124692 29690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:27.126816 29690 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } }
I20260927 22:15:27.127313 29690 leader_election.cc:302] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ad41d3cacc41819523b9511047f47e; no voters:
I20260927 22:15:27.127818 29690 leader_election.cc:288] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:27.128005 29695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:27.129217 29695 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [term 1 LEADER]: Becoming Leader. State: Replica: b9ad41d3cacc41819523b9511047f47e, State: Running, Role: LEADER
I20260927 22:15:27.129738 29695 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } }
I20260927 22:15:27.130064 29690 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:27.132751 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ad41d3cacc41819523b9511047f47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } } }
I20260927 22:15:27.132736 29697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ad41d3cacc41819523b9511047f47e. Latest consensus state: current_term: 1 leader_uuid: "b9ad41d3cacc41819523b9511047f47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ad41d3cacc41819523b9511047f47e" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 36597 } } }
I20260927 22:15:27.133172 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:27.133209 29697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:27.134776 29702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:27.140120 29702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:27.149292 29702 catalog_manager.cc:1347] Generated new cluster ID: 85fc44c0ccf44ca1bf21f13761f69edc
I20260927 22:15:27.149458 29702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:27.168678 29702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:27.170104 29702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:27.180673 29702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ad41d3cacc41819523b9511047f47e: Generated new TSK 0
I20260927 22:15:27.181466 29702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:15:27.208344 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:27.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.23.173.126:36597)
0927 22:15:27.199959 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:27.199969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:27.200448 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:15:27.200692 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:27.200696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:27.200765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:27.200883 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:15:27.201456 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.202315 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.204309 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.204748 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.204838 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.205394 (+ 556us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:15:27.207749 (+ 2355us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:15:27.208057 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:15:27.208463 29693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:15:27.210532 29719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:27.199710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:36597 (local address 127.0.0.1:38772)
0927 22:15:27.200334 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:27.200356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:27.200517 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:27.200933 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:27.200939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:27.200953 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:15:27.201306 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:27.201321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.202457 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.202462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:27.204161 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:27.204175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.204927 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.204933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:27.205113 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.205158 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:15:27.207971 (+ 2813us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:15:27.210080 (+ 2109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.126:36597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33407/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":1783,"thread_start_us":330,"threads_started":1}
I20260927 22:15:27.211619 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29624
I20260927 22:15:27.212668 29685 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:15:27.220353 29720 webserver.cc:533] Webserver: sslize failed:
E20260927 22:15:27.220472 29720 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:15:27.304292 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29624
2026-09-27T22:15:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (720 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:27Z Disabled control of system clock
I20260927 22:15:27.386119 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39447
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:39447
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/testchainca.cert with env {}
W20260927 22:15:27.709241 29725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:27.709555 29725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:27.709642 29725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:27.716917 29725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:27.717049 29725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:27.717087 29725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:27.717139 29725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:27.717183 29725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:15:27.717238 29725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:15:27.717278 29725 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:15:27.727353 29725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:39447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:27.728631 29725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:27.729910 29725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:27.742206 29733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:27.742238 29731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:27.742420 29730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:27.743151 29725 server_base.cc:1034] running on GCE node
I20260927 22:15:27.743830 29725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:27.745111 29725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:27.746299 29725 hybrid_clock.cc:648] HybridClock initialized: now 1790547327746263 us; error 25 us; skew 500 ppm
I20260927 22:15:27.746663 29725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:15:27.748837 29725 webserver.cc:460] Webserver started at http://127.23.173.126:38801/ using document root <none> and password file <none>
I20260927 22:15:27.749576 29725 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:27.749687 29725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:27.750037 29725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:27.752820 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "2939cb82a5a14dbdb69cb215f6befb2f"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "ac8d6a5fa83f137f601c7fe838d9373d"
server_key_iv: "268847eae481442f09f0a2e0e0feae1a"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.753639 29725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "2939cb82a5a14dbdb69cb215f6befb2f"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "ac8d6a5fa83f137f601c7fe838d9373d"
server_key_iv: "268847eae481442f09f0a2e0e0feae1a"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.758548 29725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:15:27.761984 29739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:27.763371 29725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:15:27.763566 29725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "2939cb82a5a14dbdb69cb215f6befb2f"
format_stamp: "Formatted at 2026-09-27 22:15:27 on dist-test-slave-3wc3"
server_key: "ac8d6a5fa83f137f601c7fe838d9373d"
server_key_iv: "268847eae481442f09f0a2e0e0feae1a"
server_key_version: "encryptionkey@0"
I20260927 22:15:27.763779 29725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:27.802062 29725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:27.802924 29725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:27.803175 29725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:27.821408 29725 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:39447
I20260927 22:15:27.821427 29790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:39447 every 8 connection(s)
I20260927 22:15:27.822885 29725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:27.824712 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29725
I20260927 22:15:27.824899 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:27.825560 24245 external_mini_cluster.cc:1428] Setting key 86a74075821539554a3655c212f31d17
I20260927 22:15:27.827350 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:27.841285 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:27.827572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.23.173.126:39447)
0927 22:15:27.828587 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:27.828603 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:27.828655 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:15:27.830690 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:27.830718 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:27.831344 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:27.831790 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:27.832520 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.834568 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.835865 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.836888 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.837069 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.838262 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:27.838273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:27.838297 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:15:27.839032 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:27.839151 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:15:27.839177 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:27.839296 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:27.839674 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 22:15:27.840026 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":284,"threads_started":1}
I20260927 22:15:27.842330 29791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f: Bootstrap starting.
I20260927 22:15:27.845376 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:15:27.846205 29791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:27.847401 29791 log.cc:824] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:27.850449 29791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f: No bootstrap required, opened a new log
I20260927 22:15:27.857414 29791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } }
I20260927 22:15:27.857788 29791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:27.857892 29791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2939cb82a5a14dbdb69cb215f6befb2f, State: Initialized, Role: FOLLOWER
I20260927 22:15:27.858546 29791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } }
I20260927 22:15:27.858843 29791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:27.858954 29791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:27.859078 29791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:27.861172 29791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } }
I20260927 22:15:27.861618 29791 leader_election.cc:302] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2939cb82a5a14dbdb69cb215f6befb2f; no voters:
I20260927 22:15:27.862226 29791 leader_election.cc:288] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:27.862370 29796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:27.863071 29796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [term 1 LEADER]: Becoming Leader. State: Replica: 2939cb82a5a14dbdb69cb215f6befb2f, State: Running, Role: LEADER
I20260927 22:15:27.863648 29796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } }
I20260927 22:15:27.864387 29791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:27.866514 29798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2939cb82a5a14dbdb69cb215f6befb2f. Latest consensus state: current_term: 1 leader_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } } }
I20260927 22:15:27.866604 29797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2939cb82a5a14dbdb69cb215f6befb2f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39447 } } }
I20260927 22:15:27.866917 29798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:27.866935 29797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:27.871132 29802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:27.877002 29802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:27.885671 29802 catalog_manager.cc:1347] Generated new cluster ID: 2cdd15a7c8394033b7d18fb9d785ac38
I20260927 22:15:27.885800 29802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:27.900475 29802 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:27.901736 29802 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:27.910166 29802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2939cb82a5a14dbdb69cb215f6befb2f: Generated new TSK 0
I20260927 22:15:27.911186 29802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:15:27.961946 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:27.926567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.23.173.126:39447)
0927 22:15:27.926767 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:27.926774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:27.927310 (+ 536us) server_negotiation.cc:407] Connection header received
0927 22:15:27.927525 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:27.927531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:27.927633 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:27.927792 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:15:27.928213 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.929379 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.930739 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.931456 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.931589 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.932051 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:15:27.961199 (+ 29148us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:15:27.961611 (+ 412us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:15:27.962129 29794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:15:27.962186 29820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:27.926530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39447 (local address 127.0.0.1:59260)
0927 22:15:27.927208 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:27.927224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:27.927343 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:27.927806 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:27.927810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:27.927819 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:15:27.928054 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:27.928061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.929503 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.929506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:27.930602 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:27.930613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:27.931617 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:27.931622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:27.931741 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:27.931771 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:15:27.961517 (+ 29746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:15:27.961749 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.173.126:39447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":395,"threads_started":1}
I20260927 22:15:27.963495 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29725
I20260927 22:15:27.964578 29786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:28.059916 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29725
2026-09-27T22:15:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (734 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (427 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1476 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:30 dist-test-slave-3wc3 krb5kdc[29836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:30 dist-test-slave-3wc3 krb5kdc[29836](info): set up 2 sockets
Sep 27 22:15:30 dist-test-slave-3wc3 krb5kdc[29836](info): commencing operation
krb5kdc: starting...
W20260927 22:15:31.050719 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:31 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:31.261603 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37981
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:31.585858 29849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:31.586320 29849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:31.586483 29849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:31.593906 29849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:31.594079 29849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:31.594197 29849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:31.594295 29849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:31.594383 29849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:31.604383 29849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:31.605755 29849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:31.607098 29849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:31.619138 29855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:31.619166 29857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:31.619225 29854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:31.620257 29849 server_base.cc:1034] running on GCE node
I20260927 22:15:31.621114 29849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:31.622622 29849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:31.623862 29849 hybrid_clock.cc:648] HybridClock initialized: now 1790547331623813 us; error 58 us; skew 500 ppm
I20260927 22:15:31.624209 29849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:31 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547331, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:31.632303 29849 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:31.633839 29849 webserver.cc:460] Webserver started at http://127.23.173.126:32797/ using document root <none> and password file <none>
I20260927 22:15:31.634660 29849 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:31.634811 29849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:31.635136 29849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:31.638140 29849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "f3b78103111341dfa7533701e2f975b3"
format_stamp: "Formatted at 2026-09-27 22:15:31 on dist-test-slave-3wc3"
server_key: "7a9151dafc01b0d64e18a9cc7efa8e62"
server_key_iv: "5f8d1ce3e5118d31687274e18986de84"
server_key_version: "encryptionkey@0"
I20260927 22:15:31.638926 29849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "f3b78103111341dfa7533701e2f975b3"
format_stamp: "Formatted at 2026-09-27 22:15:31 on dist-test-slave-3wc3"
server_key: "7a9151dafc01b0d64e18a9cc7efa8e62"
server_key_iv: "5f8d1ce3e5118d31687274e18986de84"
server_key_version: "encryptionkey@0"
I20260927 22:15:31.644042 29849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:31.647395 29864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:31.648674 29849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:31.648865 29849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "f3b78103111341dfa7533701e2f975b3"
format_stamp: "Formatted at 2026-09-27 22:15:31 on dist-test-slave-3wc3"
server_key: "7a9151dafc01b0d64e18a9cc7efa8e62"
server_key_iv: "5f8d1ce3e5118d31687274e18986de84"
server_key_version: "encryptionkey@0"
I20260927 22:15:31.649065 29849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:31.663349 29849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:31.667127 29849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:31.667409 29849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:31.686008 29849 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37981
I20260927 22:15:31.686050 29915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37981 every 8 connection(s)
I20260927 22:15:31.687383 29849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:31.691215 29916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:31.691932 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29849
I20260927 22:15:31.692190 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:31.692854 24245 external_mini_cluster.cc:1428] Setting key 50bb7bf0d62b9afc643283e654d0a448
I20260927 22:15:31.705188 29916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3: Bootstrap starting.
Sep 27 22:15:31 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:31.709031 29916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:31.710397 29916 log.cc:824] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:31.713692 29916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3: No bootstrap required, opened a new log
I20260927 22:15:31.717211 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:31.694930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.23.173.126:37981)
0927 22:15:31.695729 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:31.695741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:31.695785 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:15:31.697413 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:31.697434 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:31.697831 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:31.698232 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:31.699890 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:31.701214 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:31.703134 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:31.703819 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:31.703956 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:31.708582 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:31.708594 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:31.708613 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:31.708665 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:31.712543 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:31.713226 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:31.713234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:31.713242 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:31.713446 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:31.713855 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:31.713860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:31.713863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:31.714630 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:31.714906 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:31.715290 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:15:31.715741 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260927 22:15:31.723129 29916 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } }
I20260927 22:15:31.723474 29916 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:31.723549 29916 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b78103111341dfa7533701e2f975b3, State: Initialized, Role: FOLLOWER
I20260927 22:15:31.724128 29916 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } }
I20260927 22:15:31.724366 29916 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:31.724529 29916 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:31.724740 29916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:31.727034 29916 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } }
I20260927 22:15:31.727540 29916 leader_election.cc:302] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b78103111341dfa7533701e2f975b3; no voters:
I20260927 22:15:31.728158 29916 leader_election.cc:288] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:31.728349 29921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:31.729489 29921 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [term 1 LEADER]: Becoming Leader. State: Replica: f3b78103111341dfa7533701e2f975b3, State: Running, Role: LEADER
I20260927 22:15:31.730094 29921 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } }
I20260927 22:15:31.730451 29916 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:31.732967 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b78103111341dfa7533701e2f975b3. Latest consensus state: current_term: 1 leader_uuid: "f3b78103111341dfa7533701e2f975b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } } }
I20260927 22:15:31.733017 29922 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b78103111341dfa7533701e2f975b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b78103111341dfa7533701e2f975b3" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37981 } } }
I20260927 22:15:31.733448 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:31.733469 29922 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:31.734583 29928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:31.740191 29928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:31.748991 29928 catalog_manager.cc:1347] Generated new cluster ID: b4f0e1118e0b496187718f6402a4dfd3
I20260927 22:15:31.749125 29928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:31.764030 29928 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:31.765311 29928 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:31.777148 29928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b78103111341dfa7533701e2f975b3: Generated new TSK 0
I20260927 22:15:31.778087 29928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:31.826251 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:32.157346 29942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:32.157675 29942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:32.157840 29942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:32.165141 29942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:32.165299 29942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:32.165486 29942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:32.175535 29942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:32.176862 29942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:32.178241 29942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:32.191387 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:32.191622 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:32.191884 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:32.192211 29942 server_base.cc:1034] running on GCE node
I20260927 22:15:32.192786 29942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:32.193653 29942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:32.194875 29942 hybrid_clock.cc:648] HybridClock initialized: now 1790547332194839 us; error 50 us; skew 500 ppm
I20260927 22:15:32.195194 29942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:32 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:32.198375 29942 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:32.199890 29942 webserver.cc:460] Webserver started at http://127.23.173.65:34105/ using document root <none> and password file <none>
I20260927 22:15:32.200577 29942 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:32.200727 29942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:32.201031 29942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:32.204125 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "8d82b2a82d6a404e8659130a587a281b"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "01ecd75e9a347cb4c8cbe67e25deac32"
server_key_iv: "6e6131729d06565efdd11595c1586756"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.204934 29942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "8d82b2a82d6a404e8659130a587a281b"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "01ecd75e9a347cb4c8cbe67e25deac32"
server_key_iv: "6e6131729d06565efdd11595c1586756"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.210182 29942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:15:32.213718 29957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.215094 29942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:32.215268 29942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "8d82b2a82d6a404e8659130a587a281b"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "01ecd75e9a347cb4c8cbe67e25deac32"
server_key_iv: "6e6131729d06565efdd11595c1586756"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.215489 29942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:32.240271 29942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:32.244837 29942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:32.245079 29942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:32.245998 29942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:32.247251 29942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:32.247325 29942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.247407 29942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:32.247452 29942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.280454 29942 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:37891
I20260927 22:15:32.280503 30069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:37891 every 8 connection(s)
I20260927 22:15:32.281836 29942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:32.288371 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 29942
I20260927 22:15:32.288584 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:32.289216 24245 external_mini_cluster.cc:1428] Setting key 2bc6fd74b01e569ee2e1cc540ff48618
Sep 27 22:15:32 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:32.302778 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:32.284500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:60547 (local address 127.23.173.126:37981)
0927 22:15:32.284949 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:32.284953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:32.286017 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 22:15:32.287794 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:32.287799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:32.287899 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:32.288035 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:32.289695 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.290667 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.292378 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.293184 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.293333 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:32.296676 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:32.296686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:32.296694 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:32.296744 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:32.299545 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:32.300442 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:32.300445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:32.300448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:32.300558 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:32.301017 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:32.301021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:32.301023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:32.301261 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:32.301369 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:32.302343 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 22:15:32.302513 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
I20260927 22:15:32.304346 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:32.285147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37981 (local address 127.23.173.65:60547)
0927 22:15:32.285833 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:32.285875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:32.287544 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:32.288237 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:32.288245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:32.288857 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:32.289506 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:32.289525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.290823 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.290828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:32.292226 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:32.292237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.293400 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.293405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:32.293607 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:32.294310 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:32.294332 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:32.296442 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:32.299676 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:32.299688 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:32.299707 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:32.300325 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:32.300662 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:32.300669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:32.300675 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:32.300914 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:32.301390 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:32.301400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:32.301882 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 22:15:32.302260 (+ 378us) client_negotiation.cc:240] Negotiation successful
0927 22:15:32.302848 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":264,"threads_started":1}
I20260927 22:15:32.306041 30070 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37981
I20260927 22:15:32.306365 30070 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:32.307176 30070 heartbeater.cc:511] Master 127.23.173.126:37981 requested a full tablet report, sending...
I20260927 22:15:32.309341 29881 ts_manager.cc:194] Registered new tserver with Master: 8d82b2a82d6a404e8659130a587a281b (127.23.173.65:37891)
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:32.310971 29881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:60547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:32.337561 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:32.660040 30076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:32.660315 30076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:32.660413 30076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:32.667629 30076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:32.667744 30076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:32.667858 30076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:32.677659 30076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:32.678901 30076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:32.680250 30076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:32.693369 30081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:32.693388 30082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:32.694037 30084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:32.694459 30076 server_base.cc:1034] running on GCE node
I20260927 22:15:32.695083 30076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:32.696074 30076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:32.697305 30076 hybrid_clock.cc:648] HybridClock initialized: now 1790547332697243 us; error 51 us; skew 500 ppm
I20260927 22:15:32.697619 30076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:32 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:32.700918 30076 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:32.702491 30076 webserver.cc:460] Webserver started at http://127.23.173.66:35287/ using document root <none> and password file <none>
I20260927 22:15:32.703162 30076 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:32.703305 30076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:32.703615 30076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:32.706571 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "e4501c32ef1b4d18b5f8b1d58e8f32e7"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "38e62e923fb97b11246c9938ded1ef49"
server_key_iv: "2c16da619a7cfb854d12f6406bd06279"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.707345 30076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "e4501c32ef1b4d18b5f8b1d58e8f32e7"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "38e62e923fb97b11246c9938ded1ef49"
server_key_iv: "2c16da619a7cfb854d12f6406bd06279"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.712260 30076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:32.715587 30091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.716905 30076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:32.717069 30076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "e4501c32ef1b4d18b5f8b1d58e8f32e7"
format_stamp: "Formatted at 2026-09-27 22:15:32 on dist-test-slave-3wc3"
server_key: "38e62e923fb97b11246c9938ded1ef49"
server_key_iv: "2c16da619a7cfb854d12f6406bd06279"
server_key_version: "encryptionkey@0"
I20260927 22:15:32.717268 30076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:32.737452 30076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:32.741178 30076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:32.741467 30076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:32.742491 30076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:32.743814 30076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:32.743916 30076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.744071 30076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:32.744144 30076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:32.777141 30076 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:36155
I20260927 22:15:32.777273 30203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:36155 every 8 connection(s)
I20260927 22:15:32.778564 30076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:32.787967 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30076
I20260927 22:15:32.788177 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:32.788697 24245 external_mini_cluster.cc:1428] Setting key 12cc04b81593513b0e46b312f4fbc563
Sep 27 22:15:32 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:32.799403 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:32.781301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:37687 (local address 127.23.173.126:37981)
0927 22:15:32.781522 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:32.781528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:32.782797 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 22:15:32.784520 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:32.784525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:32.784593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:32.784685 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:32.786214 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.786956 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.789116 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.789716 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.789871 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:32.793742 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:32.793747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:32.793751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:32.793783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:32.796381 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:32.797103 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:32.797113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:32.797117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:32.797237 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:32.797677 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:32.797682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:32.797684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:32.797927 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:32.798138 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:32.798810 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 22:15:32.799061 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:15:32.800359 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:32.781818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37981 (local address 127.23.173.66:37687)
0927 22:15:32.782612 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:32.782657 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:32.784271 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:32.784913 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:32.784921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:32.785425 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:32.786052 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:32.786072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.787124 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.787128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:32.788979 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:32.788993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:32.789936 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:32.789940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:32.790199 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:32.790945 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:32.790981 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:32.793510 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:32.796522 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:32.796531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:32.796547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:32.796971 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:32.797351 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:32.797356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:32.797359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:32.797544 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:32.798106 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:32.798120 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:32.798452 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 22:15:32.798715 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:15:32.799252 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":345,"threads_started":1}
I20260927 22:15:32.801826 30204 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37981
I20260927 22:15:32.802172 30204 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:32.803090 30204 heartbeater.cc:511] Master 127.23.173.126:37981 requested a full tablet report, sending...
I20260927 22:15:32.804476 29881 ts_manager.cc:194] Registered new tserver with Master: e4501c32ef1b4d18b5f8b1d58e8f32e7 (127.23.173.66:36155)
I20260927 22:15:32.805279 29881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:37687
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:32.833492 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:33.155321 30209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:33.155601 30209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:33.155771 30209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:33.162981 30209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:33.163131 30209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:33.163318 30209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:33.173305 30209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:33.174787 30209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:33.176092 30209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:33.189273 30215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:33.189312 30217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:33.189366 30214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:33.189849 30209 server_base.cc:1034] running on GCE node
I20260927 22:15:33.190490 30209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:33.191457 30209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:33.192679 30209 hybrid_clock.cc:648] HybridClock initialized: now 1790547333192632 us; error 64 us; skew 500 ppm
I20260927 22:15:33.192991 30209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.196043 30209 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:33.197424 30209 webserver.cc:460] Webserver started at http://127.23.173.67:34549/ using document root <none> and password file <none>
I20260927 22:15:33.198237 30209 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:33.198383 30209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:33.198659 30209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:33.201601 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "a6bc33242c0d427c882c1a2e990c8560"
format_stamp: "Formatted at 2026-09-27 22:15:33 on dist-test-slave-3wc3"
server_key: "690b137dab0d4ca3bdfdccca38442dd5"
server_key_iv: "e7d53980cac20deef3c77bfb910d56c4"
server_key_version: "encryptionkey@0"
I20260927 22:15:33.202440 30209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "a6bc33242c0d427c882c1a2e990c8560"
format_stamp: "Formatted at 2026-09-27 22:15:33 on dist-test-slave-3wc3"
server_key: "690b137dab0d4ca3bdfdccca38442dd5"
server_key_iv: "e7d53980cac20deef3c77bfb910d56c4"
server_key_version: "encryptionkey@0"
I20260927 22:15:33.207333 30209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:15:33.210666 30224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:33.212031 30209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:33.212194 30209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "a6bc33242c0d427c882c1a2e990c8560"
format_stamp: "Formatted at 2026-09-27 22:15:33 on dist-test-slave-3wc3"
server_key: "690b137dab0d4ca3bdfdccca38442dd5"
server_key_iv: "e7d53980cac20deef3c77bfb910d56c4"
server_key_version: "encryptionkey@0"
I20260927 22:15:33.212393 30209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:33.223884 30209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:33.227663 30209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:33.227982 30209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:33.228943 30209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:33.230355 30209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:33.230453 30209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:33.230603 30209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:33.230675 30209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:33.263540 30209 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43461
I20260927 22:15:33.263583 30336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43461 every 8 connection(s)
I20260927 22:15:33.264868 30209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:33.273219 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30209
I20260927 22:15:33.273407 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:33.273849 24245 external_mini_cluster.cc:1428] Setting key 432139578127668997d7e6e0126e07ff
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:33.285825 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.267434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:55907 (local address 127.23.173.126:37981)
0927 22:15:33.267627 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.267632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.268825 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 22:15:33.270685 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.270691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.270802 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.270912 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.272418 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.273195 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.275018 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.275524 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.275673 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.279243 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.279248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.279252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.279285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.282736 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.283446 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.283451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.283454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.283621 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.284104 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.284109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.284112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.284461 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.284610 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.285257 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.285475 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:15:33.286597 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.267953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37981 (local address 127.23.173.67:55907)
0927 22:15:33.268657 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:33.268704 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:33.270420 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:33.271156 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:33.271165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:33.271669 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:33.272230 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:33.272269 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.273340 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.273347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:33.274867 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:33.274878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.275779 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.275790 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:33.276059 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.276794 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:33.276816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:33.279053 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:33.282871 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:33.282878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:33.282889 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:33.283308 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:33.283727 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:33.283730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:33.283732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:33.283977 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:33.284574 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:33.284580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:33.284884 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 22:15:33.285103 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:15:33.285481 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260927 22:15:33.288107 30337 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37981
I20260927 22:15:33.288435 30337 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:33.289255 30337 heartbeater.cc:511] Master 127.23.173.126:37981 requested a full tablet report, sending...
I20260927 22:15:33.290750 29881 ts_manager.cc:194] Registered new tserver with Master: a6bc33242c0d427c882c1a2e990c8560 (127.23.173.67:43461)
I20260927 22:15:33.291543 29881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:55907
I20260927 22:15:33.301559 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:15:33.314500 30070 heartbeater.cc:503] Master 127.23.173.126:37981 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/test-admin.keytab.
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.329502 24245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:15:33.334652 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:33.332270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.23.173.65:37891)
0927 22:15:33.332914 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.332921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.332948 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:15:33.333015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.333022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.333243 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.333386 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.334266 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49554: BlockingRecv error: recv got EOF from 127.0.0.1:49554 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":359,"threads_started":1}
W20260927 22:15:33.334690 24245 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.336542 24245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.337879 24245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:33.352375 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.339392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.23.173.65:37891)
0927 22:15:33.339588 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.339594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.339769 (+ 175us) server_negotiation.cc:407] Connection header received
0927 22:15:33.339896 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.339900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.339956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.340036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.341066 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.342103 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.343279 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.343837 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.343936 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.347268 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.347279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.347293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.347333 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.349893 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.350543 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.350550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.350552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.350682 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.350995 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.350998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.351000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.351327 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.351541 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.351794 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.352098 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:15:33.357880 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:33.356405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.23.173.65:37891)
0927 22:15:33.356599 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.356612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.356733 (+ 121us) server_negotiation.cc:407] Connection header received
0927 22:15:33.356868 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.356872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.356921 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.357012 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.357667 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49572: BlockingRecv error: recv got EOF from 127.0.0.1:49572 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:15:33.358331 24245 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc)
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.359786 24245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.361047 24245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:33.373893 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.362241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.23.173.65:37891)
0927 22:15:33.362387 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.362391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.362736 (+ 345us) server_negotiation.cc:407] Connection header received
0927 22:15:33.362851 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.362855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.362904 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.362979 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.363792 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.364490 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.365611 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.366083 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.366178 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.369468 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.369472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.369475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.369505 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.371877 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.372444 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.372448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.372450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.372573 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.372867 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.372870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.372872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.373111 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.373301 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.373520 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.373681 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:15:33.378214 30344 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:33.376348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.23.173.65:37891)
0927 22:15:33.376517 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.376524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.376869 (+ 345us) server_negotiation.cc:407] Connection header received
0927 22:15:33.377073 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.377076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.377147 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.377243 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.377996 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49588: BlockingRecv error: recv got EOF from 127.0.0.1:49588 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:15:33.378489 24245 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790547283249637-24245-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.379911 24245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:33.381191 24245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:33.394475 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.382409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.23.173.65:37891)
0927 22:15:33.382627 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.382632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.382892 (+ 260us) server_negotiation.cc:407] Connection header received
0927 22:15:33.383032 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.383035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.383083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.383160 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.384068 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.384752 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.385852 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.386300 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.386403 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.389769 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.389774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.389777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.389804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.392226 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.392781 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.392785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.392787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.392924 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.393228 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.393231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.393233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.393474 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.393692 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.393876 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.394113 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:15:33.396109 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29942
I20260927 22:15:33.407064 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.396909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.23.173.65:37891)
0927 22:15:33.397143 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.397147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.397225 (+ 78us) server_negotiation.cc:407] Connection header received
0927 22:15:33.397342 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.397346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.397397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.397495 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.398410 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.399057 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.400201 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.400602 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.400693 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.402670 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.402674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.402677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.402704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.404934 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.405478 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.405481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.405484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.405605 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.405893 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.405896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.405898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.406204 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.406394 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.406638 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.406789 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:15:33.407750 30065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:33.493811 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29942
I20260927 22:15:33.510154 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30076
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:33.525740 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.511108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.23.173.66:36155)
0927 22:15:33.511677 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.511684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.511708 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:33.511768 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.511774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.511977 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.512121 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.513057 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.514136 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.515232 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.515783 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.515875 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.519830 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.519840 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.519852 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.519888 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.522961 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.523657 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.523665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.523667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.523793 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.524191 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.524202 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.524205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.524523 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.524848 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.525159 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.525412 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260927 22:15:33.527050 30199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:33.611352 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30076
I20260927 22:15:33.628782 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30209
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[29836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:33.645138 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:33.629882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.23.173.67:43461)
0927 22:15:33.630617 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:33.630625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:33.630654 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:15:33.630714 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:33.630721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:33.630933 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:33.631059 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:33.632360 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.633353 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.634899 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:33.635444 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:33.635569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:33.639735 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:33.639746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:33.639760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:33.639793 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:33.642492 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.643165 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.643173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.643175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.643289 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:33.643671 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:33.643677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:33.643680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:33.644000 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:33.644321 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:33.644592 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:15:33.644842 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":425,"threads_started":1}
I20260927 22:15:33.646430 30332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:33.728925 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30209
I20260927 22:15:33.744876 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 29849
I20260927 22:15:33.746045 29911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:33.850306 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 29849
2026-09-27T22:15:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3892 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[30364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[30364](info): set up 2 sockets
Sep 27 22:15:33 dist-test-slave-3wc3 krb5kdc[30364](info): commencing operation
krb5kdc: starting...
W20260927 22:15:34.954716 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:35 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab.
Entry for principal oryx/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab.
I20260927 22:15:35.161942 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42035
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab
--principal=oryx/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:35.482251 30377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:35.482652 30377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:35.482815 30377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:35.489938 30377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:35.490130 30377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:35.490218 30377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:35.490324 30377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:35.490408 30377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:35.500123 30377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab
--principal=oryx/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:35.501448 30377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:35.502885 30377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:35.514775 30385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:35.514812 30382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:35.514768 30383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:35.515654 30377 server_base.cc:1034] running on GCE node
I20260927 22:15:35.516384 30377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:35.517720 30377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:35.518946 30377 hybrid_clock.cc:648] HybridClock initialized: now 1790547335518887 us; error 68 us; skew 500 ppm
I20260927 22:15:35.519291 30377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:35 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:35.522331 30377 init.cc:377] Logged in from keytab as oryx/127.23.173.126@KRBTEST.COM (short username oryx)
I20260927 22:15:35.523751 30377 webserver.cc:460] Webserver started at http://127.23.173.126:45349/ using document root <none> and password file <none>
I20260927 22:15:35.524426 30377 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:35.524573 30377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:35.524885 30377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:35.527904 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "9d298106e68b465aa6445ed4cdfb802f"
format_stamp: "Formatted at 2026-09-27 22:15:35 on dist-test-slave-3wc3"
server_key: "4acea82911cb76ab0c482a1e61d52755"
server_key_iv: "4de979d56a998fa96992406a0fb09d03"
server_key_version: "encryptionkey@0"
I20260927 22:15:35.528689 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "9d298106e68b465aa6445ed4cdfb802f"
format_stamp: "Formatted at 2026-09-27 22:15:35 on dist-test-slave-3wc3"
server_key: "4acea82911cb76ab0c482a1e61d52755"
server_key_iv: "4de979d56a998fa96992406a0fb09d03"
server_key_version: "encryptionkey@0"
I20260927 22:15:35.533681 30377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:15:35.536935 30392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:35.538215 30377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:35.538385 30377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9d298106e68b465aa6445ed4cdfb802f"
format_stamp: "Formatted at 2026-09-27 22:15:35 on dist-test-slave-3wc3"
server_key: "4acea82911cb76ab0c482a1e61d52755"
server_key_iv: "4de979d56a998fa96992406a0fb09d03"
server_key_version: "encryptionkey@0"
I20260927 22:15:35.538573 30377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:35.552358 30377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:35.556291 30377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:35.556584 30377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:35.575125 30377 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42035
I20260927 22:15:35.575147 30443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42035 every 8 connection(s)
I20260927 22:15:35.576571 30377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:35.580293 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:35.582221 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30377
I20260927 22:15:35.582444 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:35.583099 24245 external_mini_cluster.cc:1428] Setting key 60e482033be15c81266200344bff0d7f
I20260927 22:15:35.593478 30444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Bootstrap starting.
I20260927 22:15:35.596764 30444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:35.597899 30444 log.cc:824] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:15:35 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:35.600817 30444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: No bootstrap required, opened a new log
I20260927 22:15:35.606084 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:35.585195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.23.173.126:42035)
0927 22:15:35.586053 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:35.586073 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:35.586130 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:15:35.588464 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:35.588492 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:35.588965 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:35.589405 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:35.590765 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:35.592694 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:35.593930 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:35.594924 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:35.595156 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:35.599122 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:35.599130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:35.599142 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:35.599175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:35.602125 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:35.602779 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:35.602788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:35.602806 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:35.602980 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:35.603358 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:35.603365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:35.603372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:35.603882 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:35.604218 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:35.604644 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:15:35.605059 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":333,"threads_started":1}
I20260927 22:15:35.609288 30444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } }
I20260927 22:15:35.609661 30444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:35.609777 30444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d298106e68b465aa6445ed4cdfb802f, State: Initialized, Role: FOLLOWER
I20260927 22:15:35.610359 30444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } }
I20260927 22:15:35.610576 30444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:35.610718 30444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:35.610867 30444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:35.613366 30444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } }
I20260927 22:15:35.613894 30444 leader_election.cc:302] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d298106e68b465aa6445ed4cdfb802f; no voters:
I20260927 22:15:35.614533 30444 leader_election.cc:288] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:35.614698 30449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:35.615579 30449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 LEADER]: Becoming Leader. State: Replica: 9d298106e68b465aa6445ed4cdfb802f, State: Running, Role: LEADER
I20260927 22:15:35.616099 30449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } }
I20260927 22:15:35.616743 30444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:35.618943 30451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d298106e68b465aa6445ed4cdfb802f. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } }
I20260927 22:15:35.619355 30451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:35.619454 30450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } }
I20260927 22:15:35.620085 30450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:35.623409 30458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:35.627987 30458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:35.635876 30458 catalog_manager.cc:1347] Generated new cluster ID: 67f3e39cfa0641cc89d68dba2ca580b0
I20260927 22:15:35.635998 30458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:35.652010 30458 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:35.653803 30458 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:35.668422 30458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Generated new TSK 0
I20260927 22:15:35.669515 30458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
I20260927 22:15:35.722460 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:36.049007 30470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:36.049328 30470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:36.049496 30470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:36.060534 30470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:36.060768 30470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:36.060976 30470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:36.070873 30470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:36.072160 30470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:36.073465 30470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:36.086385 30476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:36.086504 30475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:36.086551 30478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:36.087033 30470 server_base.cc:1034] running on GCE node
I20260927 22:15:36.087584 30470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:36.088461 30470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:36.089658 30470 hybrid_clock.cc:648] HybridClock initialized: now 1790547336089623 us; error 28 us; skew 500 ppm
I20260927 22:15:36.090001 30470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:36 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547336, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:36.093104 30470 init.cc:377] Logged in from keytab as oryx/127.23.173.65@KRBTEST.COM (short username oryx)
I20260927 22:15:36.094630 30470 webserver.cc:460] Webserver started at http://127.23.173.65:40911/ using document root <none> and password file <none>
I20260927 22:15:36.095299 30470 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:36.095441 30470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:36.095745 30470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:36.098594 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "d6ea29dd0a4a427a9152c6193808eb27"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "720b0af59db258244fad1fb1dd314f83"
server_key_iv: "1a7537e9973e20b6972061006a8dc6f6"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.099360 30470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "d6ea29dd0a4a427a9152c6193808eb27"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "720b0af59db258244fad1fb1dd314f83"
server_key_iv: "1a7537e9973e20b6972061006a8dc6f6"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.104307 30470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:36.107682 30485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.108937 30470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:36.109118 30470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "d6ea29dd0a4a427a9152c6193808eb27"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "720b0af59db258244fad1fb1dd314f83"
server_key_iv: "1a7537e9973e20b6972061006a8dc6f6"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.109310 30470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:36.136181 30470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:36.139981 30470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:36.140277 30470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:36.141222 30470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:36.142630 30470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:36.142732 30470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.142882 30470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:36.142966 30470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.176040 30470 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:35279
I20260927 22:15:36.176085 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:35279 every 8 connection(s)
I20260927 22:15:36.177306 30470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:36.185024 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30470
I20260927 22:15:36.185231 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:36.186100 24245 external_mini_cluster.cc:1428] Setting key 582120dfb798720e6587359bf71b65a9
Sep 27 22:15:36 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547336, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:36.197522 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:36.179794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:49797 (local address 127.23.173.126:42035)
0927 22:15:36.180223 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:36.180230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:36.181261 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 22:15:36.183028 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:36.183032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:36.183099 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:36.183200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:36.184670 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.185467 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.187533 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.188024 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.188165 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:36.191521 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:36.191526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:36.191529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:36.191560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:36.194521 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:36.195281 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:36.195285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:36.195287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:36.195404 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:36.195876 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:36.195879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:36.195881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:36.196119 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:36.196243 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:36.197099 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 22:15:36.197271 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":194,"threads_started":1}
I20260927 22:15:36.199023 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:36.180429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:49797)
0927 22:15:36.181110 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:36.181149 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:36.182787 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:36.183393 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:36.183401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:36.183965 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:36.184511 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:36.184528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.185622 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.185628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:36.187393 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:36.187412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.188249 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.188257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:36.188493 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:36.189161 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:36.189184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:36.191313 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:36.194662 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:36.194669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:36.194681 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:36.195155 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:36.195509 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:36.195517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:36.195524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:36.195765 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:36.196256 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:36.196267 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:36.196712 (+ 445us) client_negotiation.cc:769] Sending connection context
0927 22:15:36.196998 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:15:36.197556 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260927 22:15:36.200551 30598 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:36.200871 30598 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:36.201578 30598 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:36.203943 30409 ts_manager.cc:194] Registered new tserver with Master: d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279)
I20260927 22:15:36.206212 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:49797
WARNING: no policy specified for oryx/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
I20260927 22:15:36.233141 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:36.561712 30604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:36.562067 30604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:36.562227 30604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:36.570008 30604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:36.570166 30604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:36.570348 30604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:36.583873 30604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:36.585515 30604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:36.587110 30604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:36.601783 30610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:36.601841 30609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:36.601843 30612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:36.602450 30604 server_base.cc:1034] running on GCE node
I20260927 22:15:36.602994 30604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:36.603870 30604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:36.605070 30604 hybrid_clock.cc:648] HybridClock initialized: now 1790547336605032 us; error 33 us; skew 500 ppm
I20260927 22:15:36.605350 30604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:36 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547336, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:36.608806 30604 init.cc:377] Logged in from keytab as oryx/127.23.173.66@KRBTEST.COM (short username oryx)
I20260927 22:15:36.610229 30604 webserver.cc:460] Webserver started at http://127.23.173.66:42941/ using document root <none> and password file <none>
I20260927 22:15:36.610867 30604 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:36.610965 30604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:36.611205 30604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:36.614308 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "d246fd58454b4b24bd6a9116051a931b"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "e19d343b046fd843892df83fb56d81b2"
server_key_iv: "fb802d163095da00bcb88ee68a60715f"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.615029 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "d246fd58454b4b24bd6a9116051a931b"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "e19d343b046fd843892df83fb56d81b2"
server_key_iv: "fb802d163095da00bcb88ee68a60715f"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.620095 30604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:15:36.623535 30619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.624838 30604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:36.624964 30604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "d246fd58454b4b24bd6a9116051a931b"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "e19d343b046fd843892df83fb56d81b2"
server_key_iv: "fb802d163095da00bcb88ee68a60715f"
server_key_version: "encryptionkey@0"
I20260927 22:15:36.625119 30604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:36.649899 30604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:36.654547 30604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:36.654793 30604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:36.655694 30604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:36.656930 30604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:36.657004 30604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.657109 30604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:36.657155 30604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:36.689761 30604 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:44833
I20260927 22:15:36.689805 30731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:44833 every 8 connection(s)
I20260927 22:15:36.691318 30604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:36.693279 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30604
I20260927 22:15:36.693457 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:36.694157 24245 external_mini_cluster.cc:1428] Setting key cbb71e112e45f269a307d2159f47ab98
Sep 27 22:15:36 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547336, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:36.713313 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:36.694020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50507 (local address 127.23.173.126:42035)
0927 22:15:36.694257 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:36.694265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:36.695557 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 22:15:36.697541 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:36.697546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:36.697632 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:36.697732 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:36.699307 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.700176 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.701762 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.702348 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.702485 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:36.706605 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:36.706614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:36.706621 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:36.706669 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:36.710190 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:36.710906 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:36.710910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:36.710912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:36.711052 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:36.711472 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:36.711476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:36.711478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:36.711821 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:36.711972 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:36.712714 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 22:15:36.712935 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:15:36.714380 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:36.694512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:50507)
0927 22:15:36.695386 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:36.695439 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:36.697275 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:36.698040 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:36.698051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:36.698584 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:36.699138 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:36.699155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.700323 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.700328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:36.701618 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:36.701630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:36.702596 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:36.702604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:36.702929 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:36.703663 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:36.703696 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:36.706369 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:36.710321 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:36.710329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:36.710341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:36.710762 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:36.711165 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:36.711171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:36.711174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:36.711348 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:36.711942 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:36.711950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:36.712308 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 22:15:36.712620 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 22:15:36.713176 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
WARNING: no policy specified for oryx/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:15:36.716034 30732 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:36.716399 30732 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:36.717104 30732 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:36.718726 30409 ts_manager.cc:194] Registered new tserver with Master: d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.67@KRBTEST.COM" created.
I20260927 22:15:36.719609 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:50507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
I20260927 22:15:36.743454 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:37.066512 30737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:37.066788 30737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:37.066879 30737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:37.074074 30737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:37.074198 30737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:37.074311 30737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:37.083793 30737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:37.084998 30737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:37.086390 30737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:37.098935 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:37.098974 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:37.099017 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:37.099726 30737 server_base.cc:1034] running on GCE node
I20260927 22:15:37.100323 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:37.101235 30737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:37.102451 30737 hybrid_clock.cc:648] HybridClock initialized: now 1790547337102407 us; error 61 us; skew 500 ppm
I20260927 22:15:37.102808 30737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:37.105888 30737 init.cc:377] Logged in from keytab as oryx/127.23.173.67@KRBTEST.COM (short username oryx)
I20260927 22:15:37.107381 30737 webserver.cc:460] Webserver started at http://127.23.173.67:34399/ using document root <none> and password file <none>
I20260927 22:15:37.108040 30737 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:37.108192 30737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:37.108489 30737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:37.111333 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "e9572de2364b423abdb0ce65deae31d5"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "43fc1ec2367b73eb631f5d4aa3c5b4e1"
server_key_iv: "7929efbc9e15868783ff844a9850fcf3"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.112120 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "e9572de2364b423abdb0ce65deae31d5"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "43fc1ec2367b73eb631f5d4aa3c5b4e1"
server_key_iv: "7929efbc9e15868783ff844a9850fcf3"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.117447 30737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:15:37.120986 30752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.122390 30737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:37.122562 30737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "e9572de2364b423abdb0ce65deae31d5"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "43fc1ec2367b73eb631f5d4aa3c5b4e1"
server_key_iv: "7929efbc9e15868783ff844a9850fcf3"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.122783 30737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:37.134622 30737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:37.138878 30737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:37.139122 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:37.139978 30737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:37.141215 30737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:37.141288 30737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.141371 30737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:37.141418 30737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.174955 30737 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:37269
I20260927 22:15:37.175000 30864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:37269 every 8 connection(s)
I20260927 22:15:37.176298 30737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:37.184334 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30737
I20260927 22:15:37.184513 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:37.185009 24245 external_mini_cluster.cc:1428] Setting key 69d634e81c5159c14935776089ef9ecb
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:37.195919 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.179008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:60327 (local address 127.23.173.126:42035)
0927 22:15:37.179219 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.179226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.180495 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 22:15:37.182304 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.182308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.182376 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.182467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.183934 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.184786 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.186637 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.187048 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.187187 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.190499 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.190503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.190507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.190537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.192938 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.193651 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.193655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.193657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.193779 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.194268 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.194272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.194275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.194516 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.194703 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.195369 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.195581 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:15:37.196888 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.179524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:60327)
0927 22:15:37.180314 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.180361 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.182059 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.182670 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.182677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.183156 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.183714 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.183730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.184935 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.184940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.186494 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.186505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.187269 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.187278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.187510 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.188172 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.188194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.190307 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.193079 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.193087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.193110 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.193504 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.193918 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.193924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.193928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.194144 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.194686 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.194693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.195007 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.195258 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.195760 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":303,"threads_started":1}
I20260927 22:15:37.198328 30865 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:37.198647 30865 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:37.199445 30865 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:37.200788 30409 ts_manager.cc:194] Registered new tserver with Master: e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:37.201558 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:60327
I20260927 22:15:37.209250 30598 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
I20260927 22:15:37.212299 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.173.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.125.keytab.
Entry for principal oryx/127.23.173.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.125.keytab.
I20260927 22:15:37.260960 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:40493
--webserver_interface=127.23.173.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.125.keytab
--principal=oryx/127.23.173.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42035,127.23.173.125:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:37.584690 30870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:37.585033 30870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:37.585196 30870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:37.592460 30870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:37.592602 30870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:37.592724 30870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:37.592820 30870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:37.592903 30870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:37.602797 30870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42035,127.23.173.125:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.125.keytab
--principal=oryx/127.23.173.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
W20260927 22:15:37.603195 30870 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.173.126:42035,127.23.173.125:40493'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:15:37.604246 30870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:37.605544 30870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:37.617540 30875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:37.617465 30876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:37.617512 30878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:37.618371 30870 server_base.cc:1034] running on GCE node
I20260927 22:15:37.619118 30870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:37.620574 30870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:37.621810 30870 hybrid_clock.cc:648] HybridClock initialized: now 1790547337621733 us; error 52 us; skew 500 ppm
I20260927 22:15:37.622161 30870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:37.625276 30870 init.cc:377] Logged in from keytab as oryx/127.23.173.125@KRBTEST.COM (short username oryx)
I20260927 22:15:37.626674 30870 webserver.cc:460] Webserver started at http://127.23.173.125:34435/ using document root <none> and password file <none>
I20260927 22:15:37.627342 30870 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:37.627449 30870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:37.627774 30870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:37.630812 30870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/instance:
uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "33eb0cf1f93eb869312aa7033da6bb76"
server_key_iv: "da3cd94b63fdd6af22ce3b70c71b4261"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.631798 30870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal/instance:
uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "33eb0cf1f93eb869312aa7033da6bb76"
server_key_iv: "da3cd94b63fdd6af22ce3b70c71b4261"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.637226 30870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:15:37.640969 30885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.642462 30870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:15:37.642652 30870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "33eb0cf1f93eb869312aa7033da6bb76"
server_key_iv: "da3cd94b63fdd6af22ce3b70c71b4261"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.642859 30870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:37.653445 30870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:37.657199 30870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:37.657462 30870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:37.675695 30870 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.125:40493
I20260927 22:15:37.675727 30936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.125:40493 every 8 connection(s)
I20260927 22:15:37.677037 30870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
I20260927 22:15:37.680851 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30870
I20260927 22:15:37.680934 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:37.689706 30937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.125@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:37.704800 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.683092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.23.173.125:40493)
0927 22:15:37.683922 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.683933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.683974 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:15:37.685691 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.685711 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.686154 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.686559 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.687957 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.689332 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.691173 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.692210 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.692389 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.697683 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.697690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.697700 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.697732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.701088 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.701711 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.701719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.701728 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.701865 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.702275 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.702281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.702287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.702751 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.703057 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.703527 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.703889 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":380,"threads_started":1}
I20260927 22:15:37.707394 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.692568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38824)
0927 22:15:37.693641 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.693677 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.693841 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.694391 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.694396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.694940 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.695175 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.695183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.696176 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.696183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.697683 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.697695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.698520 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.698525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.698711 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.699289 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.699306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.701542 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.704594 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.704612 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.704619 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.705152 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.705535 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.705540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.705544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.705681 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.706309 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.706316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.706476 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.706682 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.706985 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":377,"threads_started":1}
I20260927 22:15:37.707486 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.692502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.23.173.126:42035)
0927 22:15:37.692708 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.692717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.693773 (+ 1056us) server_negotiation.cc:407] Connection header received
0927 22:15:37.693999 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.694005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.694093 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.694200 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.695328 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.696037 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.697824 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.698321 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.698479 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.701734 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.701739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.701742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.701774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.704460 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.705271 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.705279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.705284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.705419 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.705787 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.705795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.705801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.706175 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.706324 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.706781 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.706989 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:15:37.708988 30937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.125@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
I20260927 22:15:37.719859 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.709522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54458)
0927 22:15:37.709744 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.709763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.709846 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.710144 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.710147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.710449 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.710573 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.710581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.711471 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.711474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.712398 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.712409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.713002 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.713006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.713176 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.713630 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.713645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.715463 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.718007 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.718010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.718013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.718366 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.718673 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.718676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.718679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.718796 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.719235 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.719238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.719349 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.719450 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.719640 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:15:37.720048 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.709704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.23.173.125:40493)
0927 22:15:37.709829 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.709833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.709847 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:15:37.709915 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.709918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.710038 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.710111 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.710695 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.711356 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.712504 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.712881 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.713008 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.715610 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.715614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.715618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.715652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.717861 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.718469 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.718472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.718474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.718582 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.718880 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.718883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.718885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.719144 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.719245 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.719527 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.719773 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":48}
I20260927 22:15:37.722157 30732 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
I20260927 22:15:37.725287 30937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Bootstrap starting.
I20260927 22:15:37.728436 30937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:37.729507 30937 log.cc:824] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:37.732044 30937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: No bootstrap required, opened a new log
I20260927 22:15:37.739010 30937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } }
I20260927 22:15:37.739360 30937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:37.739467 30937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5210b4a5fd456f82f0e9c984db2cc7, State: Initialized, Role: FOLLOWER
I20260927 22:15:37.740025 30937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } }
I20260927 22:15:37.740806 30943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } } }
I20260927 22:15:37.741073 30943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:37.741854 30937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:15:37.750311 30954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:15:37.750473 30954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:15:37.752916 30870 master_runner.cc:428] Detected that this master 0f5210b4a5fd456f82f0e9c984db2cc7 is joining an existing cluster
I20260927 22:15:37.753014 30870 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.173.125:40493
I20260927 22:15:37.773204 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.757088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.23.173.126:42035)
0927 22:15:37.757307 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.757317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.757844 (+ 527us) server_negotiation.cc:407] Connection header received
0927 22:15:37.758128 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.758134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.758222 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.758295 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.759814 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.760924 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.762994 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.763629 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.763733 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.766314 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.766321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.766326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.766365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.769858 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.770783 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.770791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.770795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.770941 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.771433 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.771438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.771442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.771804 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.771939 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.772572 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.772807 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:15:37.774313 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.757045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38834)
0927 22:15:37.757742 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.757764 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.757980 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.758524 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.758529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.759234 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.759634 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.759647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.761105 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.761110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.762825 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.762840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.764017 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.764023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.764196 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.764949 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.764971 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.766083 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.770057 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.770062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.770066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.770631 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.771086 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.771091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.771095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.771297 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.772015 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.772020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.772233 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.773470 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.773818 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":350,"threads_started":1}
I20260927 22:15:37.784854 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.758226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.23.173.125:40493)
0927 22:15:37.758388 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.758394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.769462 (+ 11068us) server_negotiation.cc:407] Connection header received
0927 22:15:37.769798 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.769805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.769873 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.769935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.771491 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.772285 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.773726 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.774560 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.774657 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.777942 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.777948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.777985 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.778024 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.781731 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.782569 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.782577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.782583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.782728 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.783124 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.783131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.783137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.783504 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.783643 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.784170 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.784415 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 22:15:37.785979 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.757908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54464)
0927 22:15:37.769359 (+ 11451us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.769382 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.769680 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.770163 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.770171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.770934 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.771332 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.771347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.772459 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.772462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.773569 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.773586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.775677 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.775685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.775923 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.776556 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.776583 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.777738 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.781867 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.781874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.781880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.782452 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.782852 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.782859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.782865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.783019 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.783725 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.783732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.783889 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.785143 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.785429 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11287,"thread_start_us":11554,"threads_started":1}
WARNING: no policy specified for oryx/127.23.173.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.124@KRBTEST.COM" created.
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.126@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
I20260927 22:15:37.809737 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.780585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54480)
0927 22:15:37.791500 (+ 10915us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.791575 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.791805 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.792302 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.792311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.793284 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.793582 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.793597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.795122 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.795128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.796745 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.796768 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.798084 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.798090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.798321 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.799089 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.799115 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.802168 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.806301 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.806317 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.806328 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.806990 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.807488 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.807493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.807497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.807667 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.808448 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.808458 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.808686 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.808930 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.809285 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10717,"thread_start_us":10588,"threads_started":1}
I20260927 22:15:37.809801 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.780943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.23.173.125:40493)
0927 22:15:37.785098 (+ 4155us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.785117 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.791669 (+ 6552us) server_negotiation.cc:407] Connection header received
0927 22:15:37.792006 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.792016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.792080 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.792223 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.793775 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.794932 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.796916 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.797650 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.797753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.802362 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.802371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.802380 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.802417 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.806161 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.807130 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.807135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.807147 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.807335 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.807800 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.807808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.807812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.808287 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.808462 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.809079 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.809358 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4053,"thread_start_us":9945,"threads_started":1}
I20260927 22:15:37.812645 30409 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.173.125:40493: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
member_type: NON_VOTER
last_known_addr {
host: "127.23.173.125"
port: 40493
}
attrs {
promote: true
}
}
dest_uuid: "9d298106e68b465aa6445ed4cdfb802f"
cas_config_opid_index: -1
I20260927 22:15:37.813689 30409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } }
I20260927 22:15:37.818799 30967 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0f5210b4a5fd456f82f0e9c984db2cc7 (127.23.173.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } }
I20260927 22:15:37.820226 30912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.124.keytab.
Entry for principal oryx/127.23.173.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.124.keytab.
I20260927 22:15:37.822319 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:39759
--webserver_interface=127.23.173.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.124.keytab
--principal=oryx/127.23.173.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:15:37.822938 30967 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.173.125:40493
I20260927 22:15:37.823671 30870 master.cc:537] Master@127.23.173.125:40493 shutting down...
I20260927 22:15:37.824342 30912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:15:37.829423 30967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } }
I20260927 22:15:37.829697 30967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:37.830967 30968 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:15:37.839920 30394 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 22:15:37.841233 30870 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:37.841853 30870 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:37.842228 30870 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: stopping tablet replica
W20260927 22:15:37.846931 30394 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f -> Peer 0f5210b4a5fd456f82f0e9c984db2cc7 (127.23.173.125:40493): Couldn't send request to peer 0f5210b4a5fd456f82f0e9c984db2cc7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:15:37.850423 30870 master.cc:552] Master@127.23.173.125:40493 shutdown complete.
I20260927 22:15:37.850510 30870 master_runner.cc:305] Clearing existing system tablet
I20260927 22:15:37.850929 30870 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:37.851023 30870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:37.858768 30870 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 22:15:37.860548 30974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.860910 30870 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:37.861179 30870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 22:15:37.861264 30870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "33eb0cf1f93eb869312aa7033da6bb76"
server_key_iv: "da3cd94b63fdd6af22ce3b70c71b4261"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.866287 30870 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:37.873178 30870 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:15:37.873353 30870 log.cc:1196] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:15:37.873788 30870 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Deleting consensus metadata
I20260927 22:15:37.874066 30870 master_runner.cc:315] Copying system tablet from 127.23.173.126:42035
I20260927 22:15:37.876619 30870 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: tablet copy: Beginning tablet copy session from remote peer at address 127.23.173.126:42035
I20260927 22:15:37.899962 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.877474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.23.173.126:42035)
0927 22:15:37.877672 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:37.877679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:37.884811 (+ 7132us) server_negotiation.cc:407] Connection header received
0927 22:15:37.884986 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:37.884993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:37.885109 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:37.885262 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:37.886474 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.887616 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.889465 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.890148 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.890283 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.892562 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:37.892569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:37.892573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:37.892614 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:37.896861 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.897722 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.897731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.897737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.897909 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:37.898414 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:37.898419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:37.898423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:37.898779 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:37.898940 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:37.899371 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 22:15:37.899621 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:15:37.900076 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:37.877395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38838)
0927 22:15:37.884642 (+ 7247us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:37.884666 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:37.884843 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:37.885394 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:37.885400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:37.885920 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:37.886274 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.886287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.887778 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.887784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.889288 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:37.889304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:37.890312 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:37.890319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:37.890511 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:37.891245 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:37.891269 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:37.892331 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:37.897034 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.897040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.897045 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.897566 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.898095 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:37.898100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:37.898104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:37.898287 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:37.898915 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:37.898920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:37.899097 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 22:15:37.899254 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:15:37.899593 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7081,"thread_start_us":6940,"threads_started":1}
I20260927 22:15:37.901312 30429 tablet_copy_service.cc:140] P 9d298106e68b465aa6445ed4cdfb802f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0f5210b4a5fd456f82f0e9c984db2cc7 ({username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:38838)
I20260927 22:15:37.901585 30429 tablet_copy_service.cc:161] P 9d298106e68b465aa6445ed4cdfb802f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0f5210b4a5fd456f82f0e9c984db2cc7 at {username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:38838: session id = 0f5210b4a5fd456f82f0e9c984db2cc7-00000000000000000000000000000000
I20260927 22:15:37.905910 30429 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:15:37.908223 30870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:15:37.915253 30870 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: tablet copy: Starting download of 0 data blocks...
I20260927 22:15:37.915575 30870 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: tablet copy: Starting download of 1 WAL segments...
I20260927 22:15:37.918538 30870 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:15:37.922915 30429 tablet_copy_service.cc:342] P 9d298106e68b465aa6445ed4cdfb802f: Request end of tablet copy session 0f5210b4a5fd456f82f0e9c984db2cc7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:38838
I20260927 22:15:37.923130 30429 tablet_copy_service.cc:433] P 9d298106e68b465aa6445ed4cdfb802f: ending tablet copy session 0f5210b4a5fd456f82f0e9c984db2cc7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0f5210b4a5fd456f82f0e9c984db2cc7
I20260927 22:15:37.934692 30870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:37.937318 30981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:37.937741 30982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:37.953377 30870 server_base.cc:1034] running on GCE node
W20260927 22:15:37.953562 30984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:37.954178 30870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:37.954867 30870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:37.956085 30870 hybrid_clock.cc:648] HybridClock initialized: now 1790547337956026 us; error 35 us; skew 500 ppm
I20260927 22:15:37.956326 30870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:37 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547337, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:37.958606 30870 init.cc:377] Logged in from keytab as oryx/127.23.173.125@KRBTEST.COM (short username oryx)
I20260927 22:15:37.959777 30870 webserver.cc:460] Webserver started at http://127.23.173.125:46279/ using document root <none> and password file <none>
I20260927 22:15:37.960125 30870 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:37.960222 30870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:37.963290 30870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:15:37.965128 30991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:37.965552 30870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 22:15:37.965696 30870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "0f5210b4a5fd456f82f0e9c984db2cc7"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "33eb0cf1f93eb869312aa7033da6bb76"
server_key_iv: "da3cd94b63fdd6af22ce3b70c71b4261"
server_key_version: "encryptionkey@0"
I20260927 22:15:37.965855 30870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:37.988621 30870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:37.992722 30870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:38.015882 30870 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.125:40493
I20260927 22:15:38.016026 31042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.125:40493 every 8 connection(s)
I20260927 22:15:38.016877 30870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
I20260927 22:15:38.022343 31043 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:15:38.026608 31043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Bootstrap starting.
I20260927 22:15:38.044492 31043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:15:38.044996 31043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Bootstrap complete.
I20260927 22:15:38.046067 31043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } }
I20260927 22:15:38.046350 31043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0f5210b4a5fd456f82f0e9c984db2cc7, State: Initialized, Role: LEARNER
I20260927 22:15:38.046602 31043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } }
I20260927 22:15:38.047483 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } }
I20260927 22:15:38.047895 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:15:38.047742 31043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:38.060055 31057 catalog_manager.cc:1259] Loaded cluster ID: 67f3e39cfa0641cc89d68dba2ca580b0
I20260927 22:15:38.060153 31057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: loading cluster ID for follower catalog manager: success
I20260927 22:15:38.064217 31057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: acquiring CA information for follower catalog manager: success
I20260927 22:15:38.066448 31057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:15:38.204041 30865 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
W20260927 22:15:38.218689 30971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:38.219002 30971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:38.219152 30971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:38.226562 30971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:38.226698 30971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:38.226796 30971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:38.226904 30971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:38.226990 30971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:38.236522 30971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.124.keytab
--principal=oryx/127.23.173.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:38.237798 30971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:38.239132 30971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:38.250963 31062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:38.250916 31063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:38.251442 31065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:38.251797 30971 server_base.cc:1034] running on GCE node
I20260927 22:15:38.252557 30971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:38.253836 30971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:38.255054 30971 hybrid_clock.cc:648] HybridClock initialized: now 1790547338255016 us; error 64 us; skew 500 ppm
I20260927 22:15:38.255383 30971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547338, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:38.266892 30971 init.cc:377] Logged in from keytab as oryx/127.23.173.124@KRBTEST.COM (short username oryx)
I20260927 22:15:38.268411 30971 webserver.cc:460] Webserver started at http://127.23.173.124:42281/ using document root <none> and password file <none>
I20260927 22:15:38.269057 30971 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:38.269156 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:38.269394 30971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:38.272347 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/instance:
uuid: "3d016a30e663494ba8c021e51bf3b650"
format_stamp: "Formatted at 2026-09-27 22:15:38 on dist-test-slave-3wc3"
server_key: "d86a4cc3e92b6dd2906a2602e934fef3"
server_key_iv: "3049e1073c6d764bb852c0080894eefe"
server_key_version: "encryptionkey@0"
I20260927 22:15:38.273082 30971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal/instance:
uuid: "3d016a30e663494ba8c021e51bf3b650"
format_stamp: "Formatted at 2026-09-27 22:15:38 on dist-test-slave-3wc3"
server_key: "d86a4cc3e92b6dd2906a2602e934fef3"
server_key_iv: "3049e1073c6d764bb852c0080894eefe"
server_key_version: "encryptionkey@0"
I20260927 22:15:38.277933 30971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:15:38.281917 31073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:38.283332 30971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:15:38.283516 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "3d016a30e663494ba8c021e51bf3b650"
format_stamp: "Formatted at 2026-09-27 22:15:38 on dist-test-slave-3wc3"
server_key: "d86a4cc3e92b6dd2906a2602e934fef3"
server_key_iv: "3049e1073c6d764bb852c0080894eefe"
server_key_version: "encryptionkey@0"
I20260927 22:15:38.283741 30971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:38.287452 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.280705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.23.173.125:40493)
0927 22:15:38.281258 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.281262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.281277 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:15:38.281335 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.281338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.281452 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.281580 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:38.282060 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.283316 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.285593 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.286489 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.286647 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.286762 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.287120 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.287216 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":323,"threads_started":1}
I20260927 22:15:38.287487 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.280505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54492)
0927 22:15:38.280704 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.280725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.280858 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.281640 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.281644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.281721 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:38.281872 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.281880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.283476 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.283482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.285447 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.285465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.286665 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.286671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.286900 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.286978 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.287077 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.287222 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:15:38.288834 31018 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:15:38.289430 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d298106e68b465aa6445ed4cdfb802f. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } }
I20260927 22:15:38.289688 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:15:38.306093 30971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:38.310401 30971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:38.310702 30971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:38.329319 30971 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.124:39759
I20260927 22:15:38.329322 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.124:39759 every 8 connection(s)
I20260927 22:15:38.330775 30971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
I20260927 22:15:38.333014 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 30971
I20260927 22:15:38.335099 31125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:38.342619 31125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547338, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.124@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:38.356348 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.335206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.23.173.124:39759)
0927 22:15:38.335898 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.335909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.335949 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:15:38.337599 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.337620 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.338049 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.338453 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.339953 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.341272 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.342705 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.343441 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.343647 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.348692 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.348701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.348711 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.348748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.352190 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.352978 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.352984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.352990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.353138 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.353580 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.353583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.353585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.354147 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.354463 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.354797 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.355225 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":318,"threads_started":1}
I20260927 22:15:38.360044 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.345126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.23.173.126:42035)
0927 22:15:38.345288 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.345293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.345988 (+ 695us) server_negotiation.cc:407] Connection header received
0927 22:15:38.346210 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.346216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.346323 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.346439 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.347503 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.348344 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.349919 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.350477 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.350604 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.354124 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.354131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.354135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.354172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.357072 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.357761 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.357768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.357772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.357896 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.358335 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.358356 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.358359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.358689 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.358839 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.359484 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.359721 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:15:38.360085 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.345089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38848)
0927 22:15:38.345833 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.345872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.346067 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.346507 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.346514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.347114 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.347349 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.347357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.348471 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.348478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.349772 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.349787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.350682 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.350693 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.350901 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.351504 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.351534 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.353907 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.357210 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.357223 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.357229 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.357629 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.358029 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.358034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.358039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.358227 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.358908 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.358916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.359128 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.359390 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.359688 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":423,"threads_started":1}
I20260927 22:15:38.361505 31125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547338, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.124@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
I20260927 22:15:38.372762 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.362078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54504)
0927 22:15:38.362253 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.362276 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.362351 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.362663 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.362666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.362942 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.363064 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.363070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.363966 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.363969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.364867 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.364876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.365493 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.365497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.365612 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.366216 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.366232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.368194 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.370895 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.370898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.370900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.371253 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.371548 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.371551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.371553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.371690 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.372154 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.372157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.372262 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.372361 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.372543 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:15:38.372848 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.362250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.23.173.125:40493)
0927 22:15:38.362420 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.362424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.362438 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:15:38.362485 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.362489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.362545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.362633 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.363180 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.363861 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.364988 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.365375 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.365465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.368370 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.368375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.368379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.368410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.370784 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.371349 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.371352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.371354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.371461 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.371774 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.371777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.371779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.372028 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.372175 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.372441 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.372601 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:15:38.373721 31125 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547338, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.124@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
I20260927 22:15:38.385231 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.374419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.0.0.1:38128)
0927 22:15:38.374715 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.374729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.374846 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.375122 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.375126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.375458 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.375586 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.375593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.376480 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.376488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.377287 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.377296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.377925 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.377930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.378137 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.378596 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.378619 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.380394 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.382926 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.382934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.382938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.383271 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.383606 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.383611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.383615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.383759 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.384256 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.384261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.384408 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.384538 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.384774 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260927 22:15:38.385306 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.374416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.23.173.124:39759)
0927 22:15:38.374593 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.374600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.374793 (+ 193us) server_negotiation.cc:407] Connection header received
0927 22:15:38.374950 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.374957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.375009 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.375100 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.375707 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.376362 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.377410 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.377784 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.377912 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.380549 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.380554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.380558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.380582 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.382805 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.383378 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.383386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.383389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.383507 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.383867 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.383871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.383874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.384102 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.384289 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.384634 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.384935 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":72}
I20260927 22:15:38.390326 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Bootstrap starting.
I20260927 22:15:38.393363 31125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:38.394861 31125 log.cc:824] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:38.398797 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: No bootstrap required, opened a new log
I20260927 22:15:38.410235 31125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } }
I20260927 22:15:38.410681 31125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:38.410794 31125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d016a30e663494ba8c021e51bf3b650, State: Initialized, Role: FOLLOWER
I20260927 22:15:38.411446 31125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } }
I20260927 22:15:38.412678 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } } }
I20260927 22:15:38.413245 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:38.414669 31125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:15:38.425184 31142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:15:38.425330 31142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:15:38.430773 30971 master_runner.cc:428] Detected that this master 3d016a30e663494ba8c021e51bf3b650 is joining an existing cluster
I20260927 22:15:38.430943 30971 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.173.124:39759
I20260927 22:15:38.455245 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.436101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.23.173.124:39759)
0927 22:15:38.436419 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.436425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.437545 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 22:15:38.437750 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.437755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.437845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.437984 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.439596 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.440870 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.443094 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.443925 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.444034 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.446974 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.446982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.446987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.447022 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.451427 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.452736 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.452743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.452747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.452927 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.453370 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.453375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.453378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.453746 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.453906 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.454560 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.454824 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 22:15:38.456305 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.436584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.0.0.1:38144)
0927 22:15:38.437396 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.437421 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.437592 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.438179 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.438184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.438967 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.439401 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.439416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.441107 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.441118 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.442935 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.442951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.444443 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.444452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.444666 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.445457 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.445509 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.446780 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.451825 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.451835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.451839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.452590 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.453069 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.453075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.453079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.453248 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.454006 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.454011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.454195 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.455452 (+ 1257us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.455759 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":398,"threads_started":1}
I20260927 22:15:38.462546 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30470
I20260927 22:15:38.467379 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.435912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.23.173.126:42035)
0927 22:15:38.436084 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.436092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.449225 (+ 13133us) server_negotiation.cc:407] Connection header received
0927 22:15:38.449397 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.449404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.449491 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.449631 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.452863 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.454027 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.456853 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.457542 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.457652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.460107 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.460113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.460116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.460147 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.463454 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.464323 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.464328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.464331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.464523 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.465042 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.465050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.465056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.465510 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.465672 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.466522 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.466832 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:15:38.467536 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.450021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.23.173.125:40493)
0927 22:15:38.450233 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.450241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.451198 (+ 957us) server_negotiation.cc:407] Connection header received
0927 22:15:38.451405 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.451413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.451511 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.451626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.453056 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.454112 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.456246 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.456916 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.457003 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.459770 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.459778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.459784 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.459827 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.463914 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.464777 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.464783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.464792 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.464960 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.465507 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.465513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.465517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.465900 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.466061 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.466820 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.467078 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:15:38.468559 31150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.450428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:54512)
0927 22:15:38.451107 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.451126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.451247 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.451740 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.451747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.452518 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.452891 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.452902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.454270 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.454276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.456078 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.456094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.457290 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.457296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.457472 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.458259 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.458280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.459554 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.464089 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.464095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.464099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.464613 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.465128 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.465133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.465137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.465335 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.466184 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.466192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.466512 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.467710 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.468028 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":63,"thread_start_us":352,"threads_started":1}
I20260927 22:15:38.470530 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.436660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38864)
0927 22:15:38.449122 (+ 12462us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.449143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.449275 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.450977 (+ 1702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.450981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.452237 (+ 1256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.452655 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.452669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.454243 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.454249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.456356 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.456707 (+ 351us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.458001 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.458009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.458260 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.458830 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.458851 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.459892 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.463665 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.463670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.463673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.464178 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.464698 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.464702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.464705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.464882 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.465777 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.465785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.466091 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.468252 (+ 2161us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.469424 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12259,"mutex_wait_us":41,"thread_start_us":12675,"threads_started":1}
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.65@KRBTEST.COM
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.126@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
I20260927 22:15:38.490873 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.464245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.23.173.65:35279)
0927 22:15:38.471943 (+ 7698us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.471950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.471974 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:38.472026 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.472032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.472354 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.472517 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.473867 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.475679 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.477086 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.477650 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.477793 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.482816 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.482835 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.482855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.482905 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.487207 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.488083 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.488096 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.488100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.488327 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.488801 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.488805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.488829 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.489293 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.489579 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.490016 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.490470 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7529,"thread_start_us":7363,"threads_started":1}
I20260927 22:15:38.491742 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.476078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.0.0.1:38156)
0927 22:15:38.476271 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.476292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.476411 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.477101 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.477107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.477589 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.477804 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.477819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.479563 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.479569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.481121 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.481137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.482200 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.482207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.482401 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.482997 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.483013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.484921 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.489051 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.489055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.489059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.489535 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.489962 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.489966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.489996 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.490159 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.490829 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.490836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.490996 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.491155 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.491392 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:15:38.492460 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.476126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.23.173.124:39759)
0927 22:15:38.476289 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.476296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.476543 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:15:38.476636 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.476642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.476738 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.476835 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.478042 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.479283 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.481296 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.482039 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.482177 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.485169 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.485179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.485185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.485238 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.488901 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.489673 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.489678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.489682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.489831 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.490304 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.490310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.490313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.490642 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.490756 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.492023 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.492252 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:15:38.492511 30593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:38.494055 30409 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.173.124:39759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3d016a30e663494ba8c021e51bf3b650"
member_type: NON_VOTER
last_known_addr {
host: "127.23.173.124"
port: 39759
}
attrs {
promote: true
}
}
dest_uuid: "9d298106e68b465aa6445ed4cdfb802f"
cas_config_opid_index: 5
I20260927 22:15:38.495040 30409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } }
I20260927 22:15:38.496902 30967 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.499205 31100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:38.500995 30967 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.173.124:39759
I20260927 22:15:38.503341 31100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:15:38.503471 31153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.504105 30967 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:38.504204 31018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:15:38.504731 30971 master.cc:537] Master@127.23.173.124:39759 shutting down...
I20260927 22:15:38.505390 31153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:38.508388 31153 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 22:15:38.526980 30393 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f -> Peer 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124:39759): Couldn't send request to peer 3d016a30e663494ba8c021e51bf3b650. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:15:38.528913 31018 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.533635 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: NON_VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: true } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.533933 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:15:38.534418 31154 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: attempting to promote NON_VOTER 0f5210b4a5fd456f82f0e9c984db2cc7 to VOTER
I20260927 22:15:38.535387 31154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } }
I20260927 22:15:38.538690 31017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 LEARNER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260927 22:15:38.538690 30393 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f -> Peer 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124:39759): Couldn't send request to peer 3d016a30e663494ba8c021e51bf3b650. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:15:38.539245 31154 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:15:38.541728 31153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0f5210b4a5fd456f82f0e9c984db2cc7 (127.23.173.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.544090 31017 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0f5210b4a5fd456f82f0e9c984db2cc7 (127.23.173.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.548818 30971 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:38.549256 30971 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:38.549489 30971 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: stopping tablet replica
I20260927 22:15:38.552982 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.553254 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:38.557631 30967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.557931 30967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:38.558221 30967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.558476 30967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:38.567615 30971 master.cc:552] Master@127.23.173.124:39759 shutdown complete.
I20260927 22:15:38.567698 30971 master_runner.cc:305] Clearing existing system tablet
I20260927 22:15:38.568084 30971 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:38.568190 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:38.571061 30971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:15:38.572561 31161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:38.572885 30971 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:38.573156 30971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:15:38.573246 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "3d016a30e663494ba8c021e51bf3b650"
format_stamp: "Formatted at 2026-09-27 22:15:38 on dist-test-slave-3wc3"
server_key: "d86a4cc3e92b6dd2906a2602e934fef3"
server_key_iv: "3049e1073c6d764bb852c0080894eefe"
server_key_version: "encryptionkey@0"
I20260927 22:15:38.578166 30971 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:38.584889 30971 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:15:38.585028 30971 log.cc:1196] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:15:38.585486 30971 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Deleting consensus metadata
I20260927 22:15:38.585777 30971 master_runner.cc:315] Copying system tablet from 127.23.173.126:42035
I20260927 22:15:38.603467 30971 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: tablet copy: Beginning tablet copy session from remote peer at address 127.23.173.126:42035
I20260927 22:15:38.616308 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.604279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.23.173.126:42035)
0927 22:15:38.604545 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.604555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.605093 (+ 538us) server_negotiation.cc:407] Connection header received
0927 22:15:38.605328 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.605334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.605411 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.605518 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:38.606525 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.607272 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.608496 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.608932 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.609063 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.610965 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:38.610971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:38.610976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:38.611016 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:38.613670 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.614330 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.614337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.614340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.614478 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:38.614856 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:38.614862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:38.614866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:38.615115 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:38.615275 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.615701 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.615933 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:15:38.616743 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.604279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:38874)
0927 22:15:38.604970 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.604990 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.605158 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.605544 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.605549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.606088 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:38.606370 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.606380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.607415 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.607422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.608342 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.608353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.609075 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.609081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.609253 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.610018 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:38.610040 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:38.610775 (+ 735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:38.613814 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.613818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.613820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.614220 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.614610 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:38.614614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:38.614616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:38.614743 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:38.615228 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:38.615233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:38.615391 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.615556 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.615897 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":428,"threads_started":1}
I20260927 22:15:38.617334 30429 tablet_copy_service.cc:140] P 9d298106e68b465aa6445ed4cdfb802f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3d016a30e663494ba8c021e51bf3b650 ({username='oryx', principal='oryx/127.23.173.124@KRBTEST.COM'} at 127.0.0.1:38874)
I20260927 22:15:38.617520 30429 tablet_copy_service.cc:161] P 9d298106e68b465aa6445ed4cdfb802f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3d016a30e663494ba8c021e51bf3b650 at {username='oryx', principal='oryx/127.23.173.124@KRBTEST.COM'} at 127.0.0.1:38874: session id = 3d016a30e663494ba8c021e51bf3b650-00000000000000000000000000000000
I20260927 22:15:38.620002 30429 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:15:38.621500 30971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:15:38.626688 30971 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: tablet copy: Starting download of 0 data blocks...
I20260927 22:15:38.626952 30971 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: tablet copy: Starting download of 1 WAL segments...
I20260927 22:15:38.629096 30971 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:15:38.632449 30429 tablet_copy_service.cc:342] P 9d298106e68b465aa6445ed4cdfb802f: Request end of tablet copy session 3d016a30e663494ba8c021e51bf3b650-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.173.124@KRBTEST.COM'} at 127.0.0.1:38874
I20260927 22:15:38.632573 30429 tablet_copy_service.cc:433] P 9d298106e68b465aa6445ed4cdfb802f: ending tablet copy session 3d016a30e663494ba8c021e51bf3b650-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3d016a30e663494ba8c021e51bf3b650
I20260927 22:15:38.643077 30971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:38.645632 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:38.646145 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:38.648463 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:38.648494 30971 server_base.cc:1034] running on GCE node
I20260927 22:15:38.649065 30971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:38.649650 30971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:38.650844 30971 hybrid_clock.cc:648] HybridClock initialized: now 1790547338650747 us; error 82 us; skew 500 ppm
I20260927 22:15:38.651155 30971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:38 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547338, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:38.653136 30971 init.cc:377] Logged in from keytab as oryx/127.23.173.124@KRBTEST.COM (short username oryx)
I20260927 22:15:38.655273 30971 webserver.cc:460] Webserver started at http://127.23.173.124:33711/ using document root <none> and password file <none>
I20260927 22:15:38.655594 30971 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:38.655709 30971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:38.658023 30971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:38.659713 31178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:38.660130 30971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 22:15:38.660259 30971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "3d016a30e663494ba8c021e51bf3b650"
format_stamp: "Formatted at 2026-09-27 22:15:38 on dist-test-slave-3wc3"
server_key: "d86a4cc3e92b6dd2906a2602e934fef3"
server_key_iv: "3049e1073c6d764bb852c0080894eefe"
server_key_version: "encryptionkey@0"
I20260927 22:15:38.660444 30971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:38.675143 30971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:38.678586 30971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:38.694315 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30470
I20260927 22:15:38.714926 30971 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.124:39759
I20260927 22:15:38.714960 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.124:39759 every 8 connection(s)
I20260927 22:15:38.715857 30971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
I20260927 22:15:38.718756 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:35279
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:42035
--webserver_port=40911
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:15:38.722528 31230 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:15:38.726912 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Bootstrap starting.
I20260927 22:15:38.741472 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:15:38.741921 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: Bootstrap complete.
I20260927 22:15:38.743001 31230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } }
I20260927 22:15:38.743402 31230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d016a30e663494ba8c021e51bf3b650, State: Initialized, Role: LEARNER
I20260927 22:15:38.743690 31230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } }
I20260927 22:15:38.744475 31234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.744822 31230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:38.745028 31234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:15:38.753638 31245 catalog_manager.cc:1259] Loaded cluster ID: 67f3e39cfa0641cc89d68dba2ca580b0
I20260927 22:15:38.753746 31245 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: loading cluster ID for follower catalog manager: success
I20260927 22:15:38.756174 31245 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: acquiring CA information for follower catalog manager: success
I20260927 22:15:38.758128 31245 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:15:38.983189 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.976098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.23.173.124:39759)
0927 22:15:38.976644 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:38.976650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:38.976671 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:15:38.976733 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:38.976736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:38.976807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:38.976907 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:38.977448 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.978531 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.981289 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.982210 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.982364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.982496 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:38.982810 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:15:38.982909 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260927 22:15:38.983188 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:38.975894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.0.0.1:38160)
0927 22:15:38.976071 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:38.976092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:38.976226 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:38.976981 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:38.976989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:38.977028 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:38.977257 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.977277 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.978690 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.978698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.981040 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:38.981063 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:38.982404 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:38.982411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:38.982639 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:38.982678 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:15:38.982780 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:15:38.982921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 22:15:38.984258 31205 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 22:15:38.984853 31234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d298106e68b465aa6445ed4cdfb802f. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:38.985172 31234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: LEARNER
W20260927 22:15:39.105070 31232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:39.105370 31232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:39.105460 31232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:39.112607 31232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:39.112697 31232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:39.112835 31232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:39.122928 31232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:35279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=40911
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:39.124396 31232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:39.125768 31232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:39.139288 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:39.139249 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:39.139325 31251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:39.140425 31232 server_base.cc:1034] running on GCE node
I20260927 22:15:39.141134 31232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:39.142143 31232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:39.143422 31232 hybrid_clock.cc:648] HybridClock initialized: now 1790547339143356 us; error 49 us; skew 500 ppm
I20260927 22:15:39.143815 31232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:39.147531 31232 init.cc:377] Logged in from keytab as oryx/127.23.173.65@KRBTEST.COM (short username oryx)
I20260927 22:15:39.149034 31232 webserver.cc:460] Webserver started at http://127.23.173.65:40911/ using document root <none> and password file <none>
I20260927 22:15:39.149693 31232 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:39.149844 31232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:39.155179 31232 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:15:39.157789 31261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.159107 31232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:39.159291 31232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "d6ea29dd0a4a427a9152c6193808eb27"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "720b0af59db258244fad1fb1dd314f83"
server_key_iv: "1a7537e9973e20b6972061006a8dc6f6"
server_key_version: "encryptionkey@0"
I20260927 22:15:39.160238 31232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:39.175137 31232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:39.179566 31232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:39.179797 31232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:39.180641 31232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:39.182140 31232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:39.182265 31232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.182382 31232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:39.182474 31232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.215718 31232 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:35279
I20260927 22:15:39.215770 31373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:35279 every 8 connection(s)
I20260927 22:15:39.217136 31232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:39.225754 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 31232
I20260927 22:15:39.226173 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30604
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:39.254009 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.221449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:51303 (local address 127.23.173.124:39759)
0927 22:15:39.221710 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.221719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.222162 (+ 443us) server_negotiation.cc:407] Connection header received
0927 22:15:39.231858 (+ 9696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.231864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.231976 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.232099 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.235865 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.237313 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.239973 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.240642 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.240742 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.245892 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.245899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.245903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.245940 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.250103 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.251098 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.251104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.251108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.251283 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.251888 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.251893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.251896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.252235 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.252383 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.253391 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.253627 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.66@KRBTEST.COM
I20260927 22:15:39.255832 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.222822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:39043 (local address 127.23.173.125:40493)
0927 22:15:39.223418 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.223427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.233556 (+ 10129us) server_negotiation.cc:407] Connection header received
0927 22:15:39.233722 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.233727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.233816 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.233929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.239195 (+ 5266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.240313 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.242639 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.243305 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.243403 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.248348 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.248355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.248359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.248399 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.252241 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.253097 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.253103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.253106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.253271 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.253678 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.253683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.253686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.254048 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.254509 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.255298 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.255509 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":273,"threads_started":1}
I20260927 22:15:39.256287 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.221307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.23.173.65:51303)
0927 22:15:39.222020 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.222064 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.231468 (+ 9404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.233678 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.233690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.234627 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.235609 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.235637 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.237529 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.237536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.239816 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.239829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.241092 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.241099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.241497 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.242332 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.242364 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.245634 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.250273 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.250285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.250304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.250953 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.251478 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.251482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.251486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.251750 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.252475 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.252484 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.252951 (+ 467us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.254281 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.254824 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":79,"spinlock_wait_cycles":12672,"thread_start_us":1271,"threads_started":1}
I20260927 22:15:39.256844 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.220574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:52977)
0927 22:15:39.222024 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.222066 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.233030 (+ 10964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.237868 (+ 4838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.237873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.238576 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.239165 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.239181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.240748 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.240753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.242745 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.242758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.243964 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.243970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.244181 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.245897 (+ 1716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.245920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.249774 (+ 3854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.253865 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.253870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.253874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.254433 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.254886 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.254891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.254895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.255072 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.255736 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.255744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.255928 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.256063 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.256372 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"spinlock_wait_cycles":17792,"thread_start_us":854,"threads_started":1}
I20260927 22:15:39.257169 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.222552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.23.173.65:39043)
0927 22:15:39.233457 (+ 10905us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.233480 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.233614 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.237388 (+ 3774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.237392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.238279 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.239012 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.239029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.240490 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.240495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.242469 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.242482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.243722 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.243730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.243942 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.244527 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.244548 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.248091 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.252385 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.252390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.252395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.252951 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.253409 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.253413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.253416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.253553 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.254586 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.254589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.254728 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.254865 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.255099 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10752,"mutex_wait_us":4,"thread_start_us":10595,"threads_started":1}
I20260927 22:15:39.257942 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.221053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:52977 (local address 127.23.173.126:42035)
0927 22:15:39.237369 (+ 16316us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.237378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.237402 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:39.237476 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.237484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.237579 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.238397 (+ 818us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.239384 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.240584 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.242912 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.243610 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.243731 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.250028 (+ 6297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.250035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.250039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.250083 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.253717 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.254590 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.254594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.254597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.254747 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.255232 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.255236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.255239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.255586 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.257205 (+ 1619us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.257324 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.257536 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16151,"thread_start_us":16640,"threads_started":1}
I20260927 22:15:39.258512 31374 heartbeater.cc:347] Connected to a master server at 127.23.173.124:39759
I20260927 22:15:39.258817 31374 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.259552 31374 heartbeater.cc:511] Master 127.23.173.124:39759 requested a full tablet report, sending...
I20260927 22:15:39.260437 31376 heartbeater.cc:347] Connected to a master server at 127.23.173.125:40493
I20260927 22:15:39.260562 31376 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.260999 31376 heartbeater.cc:511] Master 127.23.173.125:40493 requested a full tablet report, sending...
I20260927 22:15:39.261685 31375 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:39.261796 31375 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.262245 31375 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:39.262205 31195 ts_manager.cc:194] Registered new tserver with Master: d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279)
I20260927 22:15:39.262765 31008 ts_manager.cc:194] Registered new tserver with Master: d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279)
I20260927 22:15:39.263309 30409 ts_manager.cc:194] Re-registered known tserver with Master: d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279)
I20260927 22:15:39.263614 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.227385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.23.173.66:44833)
0927 22:15:39.240231 (+ 12846us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.240239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.240267 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:15:39.240328 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.240335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.240550 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.240718 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.242932 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.245286 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.247091 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.247903 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.248086 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.255462 (+ 7376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.255476 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.255493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.255532 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.259468 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.260296 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.260302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.260304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.260435 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.260925 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.260930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.260933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.261348 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.261654 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.262907 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.263177 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12666,"thread_start_us":14428,"threads_started":1}
I20260927 22:15:39.264312 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:52977
I20260927 22:15:39.264853 30727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:39.347779 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30604
I20260927 22:15:39.365490 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:44833
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:42035
--webserver_port=42941
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:15:39.583725 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: NON_VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: true } } }
I20260927 22:15:39.584064 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:39.585251 31158 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: attempting to promote NON_VOTER 3d016a30e663494ba8c021e51bf3b650 to VOTER
I20260927 22:15:39.585822 31158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } }
I20260927 22:15:39.587824 31205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 LEARNER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:15:39.587972 31017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Refusing update from remote peer 9d298106e68b465aa6445ed4cdfb802f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:15:39.588459 31388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:15:39.588937 31158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:15:39.591663 31158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.593775 31205 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.594803 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.595026 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:39.595408 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.595574 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:39.595993 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.596271 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:39.596544 31017 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.602453 31394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.602937 31394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:39.603780 31393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d298106e68b465aa6445ed4cdfb802f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:39.604130 31393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:15:39.694852 31386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:39.695144 31386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:39.695304 31386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:39.702711 31386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:39.702831 31386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:39.703030 31386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:39.712669 31386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:44833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=42941
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:39.714073 31386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:39.715839 31386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:39.730324 31399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:39.730366 31400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:39.730899 31402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:39.732200 31386 server_base.cc:1034] running on GCE node
I20260927 22:15:39.732700 31386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:39.733577 31386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:39.734761 31386 hybrid_clock.cc:648] HybridClock initialized: now 1790547339734702 us; error 50 us; skew 500 ppm
I20260927 22:15:39.735118 31386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:39.738181 31386 init.cc:377] Logged in from keytab as oryx/127.23.173.66@KRBTEST.COM (short username oryx)
I20260927 22:15:39.739521 31386 webserver.cc:460] Webserver started at http://127.23.173.66:42941/ using document root <none> and password file <none>
I20260927 22:15:39.740190 31386 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:39.740340 31386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:39.745805 31386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:39.748674 31409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.750089 31386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:39.750254 31386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "d246fd58454b4b24bd6a9116051a931b"
format_stamp: "Formatted at 2026-09-27 22:15:36 on dist-test-slave-3wc3"
server_key: "e19d343b046fd843892df83fb56d81b2"
server_key_iv: "fb802d163095da00bcb88ee68a60715f"
server_key_version: "encryptionkey@0"
I20260927 22:15:39.751291 31386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:39.768024 31386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:39.772302 31386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:39.772593 31386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:39.773547 31386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:39.774940 31386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:39.775051 31386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.775184 31386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:39.775288 31386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:39.807219 31386 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:44833
I20260927 22:15:39.807296 31521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:44833 every 8 connection(s)
I20260927 22:15:39.808727 31386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:39.819242 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 31386
I20260927 22:15:39.819550 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30737
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
I20260927 22:15:39.853675 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.811752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:33841 (local address 127.23.173.125:40493)
0927 22:15:39.827230 (+ 15478us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.827241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.827274 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:15:39.828794 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.828803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.828927 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.829053 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.832386 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.833887 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.837194 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.838106 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.838243 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.845474 (+ 7231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.845481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.845485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.845538 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.849478 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.850699 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.850705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.850709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.850851 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.851433 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.851438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.851442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.851836 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.851964 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.853004 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.853249 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15307,"thread_start_us":25915,"threads_started":1}
I20260927 22:15:39.854565 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.811794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38297 (local address 127.23.173.124:39759)
0927 22:15:39.830400 (+ 18606us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.830409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.832350 (+ 1941us) server_negotiation.cc:407] Connection header received
0927 22:15:39.832408 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.832411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.832472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.832590 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.833692 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.835048 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.837462 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.838287 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.838427 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.845427 (+ 7000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.845435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.845441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.845522 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.849744 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.850687 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.850693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.850696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.850851 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.851425 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.851430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.851433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.851789 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.851925 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.854079 (+ 2154us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.854268 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18448,"thread_start_us":18338,"threads_started":1}
I20260927 22:15:39.855553 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.812285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.23.173.66:38297)
0927 22:15:39.830865 (+ 18580us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.830884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.830987 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.832610 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.832613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.833195 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.833498 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.833512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.835254 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.835261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.837258 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.837276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.838696 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.838705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.839128 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.840192 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.840244 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.845178 (+ 4934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.849899 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.849912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.849915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.850547 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.850989 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.850993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.850997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.851275 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.852018 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.852029 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.852518 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.852875 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.853358 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18436,"mutex_wait_us":196,"spinlock_wait_cycles":94848,"thread_start_us":13294,"threads_started":1}
I20260927 22:15:39.855575 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.812285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.23.173.66:33841)
0927 22:15:39.822841 (+ 10556us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.822905 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.828258 (+ 5353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.829598 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.829628 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.831641 (+ 2013us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.832207 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.832224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.834143 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.834161 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.836968 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.836992 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.838674 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.838683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.839120 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.840253 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.840287 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.845177 (+ 4890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.849661 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.849671 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.849689 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.850512 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.850997 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.851000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.851004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.851308 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.852044 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.852048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.852537 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.854011 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.854310 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10131,"mutex_wait_us":314,"spinlock_wait_cycles":63104,"thread_start_us":18148,"threads_started":1}
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547339, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:39.857440 31524 heartbeater.cc:347] Connected to a master server at 127.23.173.125:40493
I20260927 22:15:39.857775 31524 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.858073 31522 heartbeater.cc:347] Connected to a master server at 127.23.173.124:39759
I20260927 22:15:39.858227 31522 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.858891 31524 heartbeater.cc:511] Master 127.23.173.125:40493 requested a full tablet report, sending...
I20260927 22:15:39.858924 31522 heartbeater.cc:511] Master 127.23.173.124:39759 requested a full tablet report, sending...
I20260927 22:15:39.860870 31195 ts_manager.cc:194] Registered new tserver with Master: d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
I20260927 22:15:39.860875 31008 ts_manager.cc:194] Registered new tserver with Master: d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
Sep 27 22:15:39 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.67@KRBTEST.COM
I20260927 22:15:39.864477 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.831144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34865 (local address 127.23.173.126:42035)
0927 22:15:39.845399 (+ 14255us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.845405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.845423 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:39.845484 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.845489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.845568 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.845699 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.847598 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.848828 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.852133 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.852786 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.852919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.857495 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.857504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.857510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.857566 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.861194 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.862544 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.862553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.862559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.862713 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.863097 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.863105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.863109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.863466 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.863644 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.863971 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.864208 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14108,"thread_start_us":13941,"threads_started":1}
I20260927 22:15:39.864480 31532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.831032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:34865)
0927 22:15:39.831662 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:39.831685 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:39.831847 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:39.846303 (+ 14456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:39.846308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:39.846929 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:39.847273 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.847284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.849674 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.849681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.851381 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:39.851396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.852978 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.852985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:39.853187 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.853829 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:39.853851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:39.856847 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:39.861827 (+ 4980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.861832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.861837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.862401 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.862830 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:39.862833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:39.862836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:39.862969 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:39.863579 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:39.863583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:39.863702 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 22:15:39.863841 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:15:39.864049 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":302,"threads_started":1}
I20260927 22:15:39.865505 31523 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:39.865644 31523 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:39.866135 31523 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:39.867187 30409 ts_manager.cc:194] Re-registered known tserver with Master: d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
I20260927 22:15:39.868187 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:34865
I20260927 22:15:39.870936 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:39.820948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.23.173.67:37269)
0927 22:15:39.845710 (+ 24762us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:39.845722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:39.847005 (+ 1283us) server_negotiation.cc:407] Connection header received
0927 22:15:39.847083 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:39.847093 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:39.847394 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:39.848079 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:39.850665 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.852135 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.857001 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:39.857639 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:39.857729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:39.864257 (+ 6528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:39.864273 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:39.864290 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:39.864338 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:39.867922 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.868743 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.868750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.868752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.868919 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:39.869282 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:39.869287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:39.869291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:39.869704 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:39.870033 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:39.870294 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:15:39.870548 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24546,"thread_start_us":24849,"threads_started":1}
I20260927 22:15:39.872062 30860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:39.955765 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30737
I20260927 22:15:39.972788 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:37269
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:42035
--webserver_port=34399
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:15:40.267787 31375 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
W20260927 22:15:40.310585 31535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:40.310844 31535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:40.310936 31535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:40.318336 31535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:40.318480 31535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:40.318645 31535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:40.328173 31535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:37269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=34399
--tserver_master_addrs=127.23.173.126:42035,127.23.173.125:40493,127.23.173.124:39759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:40.329497 31535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:40.330835 31535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:40.343500 31540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:40.343573 31541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:40.343540 31543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:40.344578 31535 server_base.cc:1034] running on GCE node
I20260927 22:15:40.345232 31535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:40.346258 31535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:40.347492 31535 hybrid_clock.cc:648] HybridClock initialized: now 1790547340347451 us; error 53 us; skew 500 ppm
I20260927 22:15:40.347892 31535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:40 dist-test-slave-3wc3 krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547340, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:40.351213 31535 init.cc:377] Logged in from keytab as oryx/127.23.173.67@KRBTEST.COM (short username oryx)
I20260927 22:15:40.352626 31535 webserver.cc:460] Webserver started at http://127.23.173.67:34399/ using document root <none> and password file <none>
I20260927 22:15:40.353302 31535 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:40.353456 31535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:40.358939 31535 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:15:40.361675 31550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:40.362993 31535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:40.363171 31535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "e9572de2364b423abdb0ce65deae31d5"
format_stamp: "Formatted at 2026-09-27 22:15:37 on dist-test-slave-3wc3"
server_key: "43fc1ec2367b73eb631f5d4aa3c5b4e1"
server_key_iv: "7929efbc9e15868783ff844a9850fcf3"
server_key_version: "encryptionkey@0"
I20260927 22:15:40.364256 31535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:40.382540 31535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:40.386281 31535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:40.386561 31535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:40.387508 31535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:40.388803 31535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:40.388917 31535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:40.389048 31535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:40.389165 31535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:40.422135 31535 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:37269
I20260927 22:15:40.422310 31662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:37269 every 8 connection(s)
I20260927 22:15:40.423607 31535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:40.433321 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 31535
Sep 27 22:15:40 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547340, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
Sep 27 22:15:40 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547340, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.124@KRBTEST.COM
Sep 27 22:15:40 dist-test-slave-3wc3 krb5kdc[30364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547340, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.125@KRBTEST.COM
I20260927 22:15:40.464017 31668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.427325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34437 (local address 127.23.173.124:39759)
0927 22:15:40.433069 (+ 5744us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.433075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.434268 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 22:15:40.434514 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.434520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.434608 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.434689 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.436363 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.437874 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.441085 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.442027 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.442167 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.450204 (+ 8037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.450211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.450216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.450260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.453866 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.457873 (+ 4007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.457878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.457881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.458075 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.460047 (+ 1972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.460056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.460061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.460545 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.460700 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.463343 (+ 2643us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.463643 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5623,"thread_start_us":5528,"threads_started":1}
I20260927 22:15:40.466387 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.427342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.23.173.67:34437)
0927 22:15:40.434165 (+ 6823us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.434185 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.434368 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.434914 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.434922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.435721 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:40.436148 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.436165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.438132 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.438140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.440850 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.440887 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.442624 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.442633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.443084 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.445740 (+ 2656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:40.445774 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:40.449977 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:40.456105 (+ 6128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.456111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.456116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.457281 (+ 1165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.458271 (+ 990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.458279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.458285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.458477 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.461801 (+ 3324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:40.461808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:40.462053 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.464232 (+ 2179us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.464799 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6632,"mutex_wait_us":395,"thread_start_us":2728,"threads_started":1}
I20260927 22:15:40.467818 31667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.426578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:56503 (local address 127.23.173.126:42035)
0927 22:15:40.427105 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.427111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.433133 (+ 6022us) server_negotiation.cc:407] Connection header received
0927 22:15:40.433206 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.433211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.433299 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.433422 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.436157 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.437679 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.441395 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.442352 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.442491 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.448484 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.448490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.448495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.448538 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.455442 (+ 6904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.456809 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.456814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.456818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.456994 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.459287 (+ 2293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.459295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.459301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.459751 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.459937 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.467184 (+ 7247us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.467437 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":219,"threads_started":1}
I20260927 22:15:40.468217 31663 heartbeater.cc:347] Connected to a master server at 127.23.173.124:39759
I20260927 22:15:40.468595 31663 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:40.469599 31663 heartbeater.cc:511] Master 127.23.173.124:39759 requested a full tablet report, sending...
I20260927 22:15:40.472193 31195 ts_manager.cc:194] Registered new tserver with Master: e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:40.475440 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.427342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:56503)
0927 22:15:40.430586 (+ 3244us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.430632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.432342 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.433782 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.433799 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.434723 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:40.435870 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.435902 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.437938 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.437946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.441184 (+ 3238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.441205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.442908 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.442917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.443250 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.444143 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:40.444185 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:40.448215 (+ 4030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:40.455604 (+ 7389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.455613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.455633 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.456589 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.457426 (+ 837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.457430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.457433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.457721 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.460817 (+ 3096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:40.460830 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:40.461396 (+ 566us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.468135 (+ 6739us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.468445 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2961,"mutex_wait_us":74,"spinlock_wait_cycles":19328,"thread_start_us":6357,"threads_started":1}
I20260927 22:15:40.476572 31664 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42035
I20260927 22:15:40.476706 31664 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:40.477161 31664 heartbeater.cc:511] Master 127.23.173.126:42035 requested a full tablet report, sending...
I20260927 22:15:40.478193 30409 ts_manager.cc:194] Re-registered known tserver with Master: e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:40.479288 30409 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:56503
I20260927 22:15:40.481735 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.449126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.23.173.126:42035)
0927 22:15:40.464219 (+ 15093us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.464225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.464248 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:15:40.464296 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.464300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.464365 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.464461 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.466933 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.468152 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.470109 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.470807 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.470939 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.474507 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.474513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.474517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.474555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.478009 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.478877 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.478882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.478885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.479048 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.479473 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.479477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.479480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.479823 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.480053 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.480917 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.481118 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14960,"thread_start_us":15530,"threads_started":1}
I20260927 22:15:40.482820 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.434543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:44035 (local address 127.23.173.125:40493)
0927 22:15:40.445444 (+ 10901us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.445454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.445479 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:15:40.445554 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.445561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.445660 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.445790 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.447929 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.452251 (+ 4322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.454356 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.455160 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.455273 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.468748 (+ 13475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.468757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.468762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.468815 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.472969 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.473840 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.473846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.473850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.474035 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.474486 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.474491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.474494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.474859 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.474986 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.482160 (+ 7174us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.482414 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10730,"thread_start_us":10729,"threads_started":1}
I20260927 22:15:40.483367 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.438835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.23.173.67:44035)
0927 22:15:40.445067 (+ 6232us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.445091 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.445201 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.446399 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.446408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.447235 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:40.447699 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.447713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.452446 (+ 4733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.452451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.454188 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.454204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.455592 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.455598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.455823 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.456588 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:40.456611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:40.463009 (+ 6398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:40.473154 (+ 10145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.473160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.473165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.473686 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.474191 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:40.474197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:40.474201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:40.474359 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:40.475064 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:40.475071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:40.475275 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.475449 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.475733 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6081,"thread_start_us":5931,"threads_started":1}
I20260927 22:15:40.484369 31665 heartbeater.cc:347] Connected to a master server at 127.23.173.125:40493
I20260927 22:15:40.484519 31665 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:40.484956 31665 heartbeater.cc:511] Master 127.23.173.125:40493 requested a full tablet report, sending...
I20260927 22:15:40.486193 31008 ts_manager.cc:194] Registered new tserver with Master: e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:40.490777 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.452577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.23.173.124:39759)
0927 22:15:40.460968 (+ 8391us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.460973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.465141 (+ 4168us) server_negotiation.cc:407] Connection header received
0927 22:15:40.465444 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.465452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.465556 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.465649 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.475950 (+ 10301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.477468 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.479774 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.480479 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.480598 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.484675 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.484684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.484691 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.484733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.487711 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.488529 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.488535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.488540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.488686 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.489126 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.489131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.489135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.489526 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.489762 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.490176 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.490412 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8260,"thread_start_us":7778,"threads_started":1}
I20260927 22:15:40.486093 30409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:40.496863 30409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:40.497751 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.462122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.23.173.125:40493)
0927 22:15:40.479650 (+ 17528us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.479657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.479676 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:15:40.479729 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.479737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.479810 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.479912 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:40.482676 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.486028 (+ 3352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.487998 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.488673 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.488797 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.491925 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:40.491932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:40.491937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:40.491974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:40.495071 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.495765 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.495770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.495773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.495917 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:40.496262 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:40.496266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:40.496269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:40.496598 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:40.496825 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.497112 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.497319 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17395,"thread_start_us":20363,"threads_started":1}
I20260927 22:15:40.544364 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.518448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:44833 (local address 127.0.0.1:56720)
0927 22:15:40.529882 (+ 11434us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.529905 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.530216 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.532113 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.532133 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.532170 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:40.532711 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.532729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.536783 (+ 4054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.536792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.539488 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.539511 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.543173 (+ 3662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.543182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.543452 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.543512 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.543597 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.543798 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11265,"threads_started":1}
I20260927 22:15:40.546110 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.518703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.23.173.66:44833)
0927 22:15:40.531289 (+ 12586us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.531304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.531343 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:15:40.531470 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.531483 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.531839 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.532993 (+ 1154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:40.533271 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.536586 (+ 3315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.539754 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.542880 (+ 3126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.544988 (+ 2108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.545356 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.545453 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.545696 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12425,"thread_start_us":12314,"threads_started":1}
I20260927 22:15:40.556042 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.519558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.23.173.65:35279)
0927 22:15:40.549089 (+ 29531us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.549101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.549140 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:15:40.549266 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.549277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.549612 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.549854 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:40.550178 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.552170 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.553709 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.554774 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.554915 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.555138 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.555573 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.555753 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29312,"thread_start_us":29184,"threads_started":1}
I20260927 22:15:40.556634 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.519271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:35279 (local address 127.0.0.1:41726)
0927 22:15:40.544646 (+ 25375us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.544669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.544793 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.549814 (+ 5021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.549817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.549837 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:40.550019 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.550029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.552305 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.552309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.553550 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.553562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.554934 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.554944 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.555206 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.555260 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.555372 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.556369 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25237,"thread_start_us":10708,"threads_started":1}
I20260927 22:15:40.559695 31457 tablet_service.cc:1467] Processing CreateTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 (DEFAULT_TABLE table=test-table [id=809661391e95420c92d554e7a552b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:40.560112 31309 tablet_service.cc:1467] Processing CreateTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 (DEFAULT_TABLE table=test-table [id=809661391e95420c92d554e7a552b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:40.561347 31457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02562d64a8a944f2a96e73ae9cee9057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:40.561435 31309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02562d64a8a944f2a96e73ae9cee9057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:40.562352 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.518880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37269 (local address 127.0.0.1:48786)
0927 22:15:40.544326 (+ 25446us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.544356 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.544974 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.546563 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.546570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.546608 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:40.547116 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.547133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.554111 (+ 6978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.554119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.556404 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.556420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.561482 (+ 5062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.561490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.561648 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.561673 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.561778 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.562010 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25252,"thread_start_us":33449,"threads_started":1}
I20260927 22:15:40.562386 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.519578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.23.173.67:37269)
0927 22:15:40.543439 (+ 23861us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.543464 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.544785 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 22:15:40.545239 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.545252 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.545623 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.545878 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:40.547371 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.553883 (+ 6512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.556622 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.561054 (+ 4432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.561200 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.561343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.561833 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.562065 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23606,"thread_start_us":23918,"threads_started":1}
I20260927 22:15:40.565598 31598 tablet_service.cc:1467] Processing CreateTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 (DEFAULT_TABLE table=test-table [id=809661391e95420c92d554e7a552b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:40.566970 31598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02562d64a8a944f2a96e73ae9cee9057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:40.583451 31697 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Bootstrap starting.
I20260927 22:15:40.583609 31698 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Bootstrap starting.
I20260927 22:15:40.587240 31697 tablet_bootstrap.cc:654] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:40.587241 31698 tablet_bootstrap.cc:654] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:40.588581 31697 log.cc:824] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:40.588580 31698 log.cc:824] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:40.591436 31699 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Bootstrap starting.
I20260927 22:15:40.593735 31697 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: No bootstrap required, opened a new log
I20260927 22:15:40.594053 31697 ts_tablet_manager.cc:1397] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 22:15:40.595384 31699 tablet_bootstrap.cc:654] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:40.596457 31698 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: No bootstrap required, opened a new log
I20260927 22:15:40.596623 31699 log.cc:824] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:40.596843 31698 ts_tablet_manager.cc:1397] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260927 22:15:40.605940 31699 tablet_bootstrap.cc:492] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: No bootstrap required, opened a new log
I20260927 22:15:40.606271 31699 ts_tablet_manager.cc:1397] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260927 22:15:40.606505 31697 raft_consensus.cc:348] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.606982 31697 raft_consensus.cc:374] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:40.607098 31697 raft_consensus.cc:729] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ea29dd0a4a427a9152c6193808eb27, State: Initialized, Role: FOLLOWER
I20260927 22:15:40.607764 31697 consensus_queue.cc:260] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.608824 31698 raft_consensus.cc:348] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.609247 31698 raft_consensus.cc:374] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:40.609333 31698 raft_consensus.cc:729] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d246fd58454b4b24bd6a9116051a931b, State: Initialized, Role: FOLLOWER
I20260927 22:15:40.609539 31697 ts_tablet_manager.cc:1428] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 22:15:40.609897 31698 consensus_queue.cc:260] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.611580 31698 ts_tablet_manager.cc:1428] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260927 22:15:40.612056 31523 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
I20260927 22:15:40.620190 31699 raft_consensus.cc:348] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.620504 31699 raft_consensus.cc:374] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:40.620631 31699 raft_consensus.cc:729] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9572de2364b423abdb0ce65deae31d5, State: Initialized, Role: FOLLOWER
I20260927 22:15:40.621235 31699 consensus_queue.cc:260] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.622397 31664 heartbeater.cc:503] Master 127.23.173.126:42035 was elected leader, sending a full tablet report...
I20260927 22:15:40.623082 31699 ts_tablet_manager.cc:1428] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Time spent starting tablet: real 0.017s user 0.009s sys 0.002s
W20260927 22:15:40.677239 31666 tablet.cc:2369] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:40.721470 31377 tablet.cc:2369] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:40.812489 31525 tablet.cc:2369] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:15:40.896670 31703 raft_consensus.cc:482] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:40.897007 31703 raft_consensus.cc:504] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.898772 31703 leader_election.cc:288] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833), e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:40.905778 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.899444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:41627 (local address 127.23.173.67:37269)
0927 22:15:40.899673 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.899682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.900324 (+ 642us) server_negotiation.cc:407] Connection header received
0927 22:15:40.900517 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.900523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.900600 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.900708 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:40.901282 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.902103 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.904060 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.904820 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.904993 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.905090 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.905386 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.905518 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:15:40.905828 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.899264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:37637 (local address 127.23.173.66:44833)
0927 22:15:40.899529 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.899538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.900257 (+ 719us) server_negotiation.cc:407] Connection header received
0927 22:15:40.900563 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.900568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.900645 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.900732 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:40.901398 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.902194 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.903996 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.904825 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.905010 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.905090 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.905441 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.905573 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:15:40.905905 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.899083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37269 (local address 127.23.173.65:41627)
0927 22:15:40.900191 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.900205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.900352 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.900749 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.900753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.900782 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:40.901105 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.901113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.902241 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.902249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.903885 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.903896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.905063 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.905069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.905220 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.905251 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.905327 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.905462 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":83,"thread_start_us":669,"threads_started":1}
I20260927 22:15:40.906147 31707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.898990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:44833 (local address 127.23.173.65:37637)
0927 22:15:40.900131 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:40.900146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:40.900333 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:40.900905 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:40.900909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:40.900930 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:40.901256 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.901266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.902330 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.902333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.903846 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:40.903858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.905131 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.905145 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:40.905263 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.905284 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:15:40.905397 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:15:40.905516 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":3,"thread_start_us":863,"threads_started":1}
I20260927 22:15:40.907051 31618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02562d64a8a944f2a96e73ae9cee9057" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9572de2364b423abdb0ce65deae31d5" is_pre_election: true
I20260927 22:15:40.907150 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02562d64a8a944f2a96e73ae9cee9057" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d246fd58454b4b24bd6a9116051a931b" is_pre_election: true
I20260927 22:15:40.907625 31618 raft_consensus.cc:2454] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 0.
I20260927 22:15:40.907636 31477 raft_consensus.cc:2454] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 0.
I20260927 22:15:40.908324 31262 leader_election.cc:302] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d246fd58454b4b24bd6a9116051a931b, d6ea29dd0a4a427a9152c6193808eb27, e9572de2364b423abdb0ce65deae31d5; no voters:
I20260927 22:15:40.908742 31703 raft_consensus.cc:2781] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:40.908890 31703 raft_consensus.cc:482] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:40.909010 31703 raft_consensus.cc:3037] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:40.911167 31703 raft_consensus.cc:504] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.911748 31703 leader_election.cc:288] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 election: Requested vote from peers d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833), e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269)
I20260927 22:15:40.912207 31618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02562d64a8a944f2a96e73ae9cee9057" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9572de2364b423abdb0ce65deae31d5"
I20260927 22:15:40.912209 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02562d64a8a944f2a96e73ae9cee9057" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d246fd58454b4b24bd6a9116051a931b"
I20260927 22:15:40.912410 31618 raft_consensus.cc:3037] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:40.912410 31477 raft_consensus.cc:3037] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:40.915673 31618 raft_consensus.cc:2454] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 1.
I20260927 22:15:40.915663 31477 raft_consensus.cc:2454] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 1.
I20260927 22:15:40.916108 31265 leader_election.cc:302] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ea29dd0a4a427a9152c6193808eb27, e9572de2364b423abdb0ce65deae31d5; no voters:
I20260927 22:15:40.916438 31703 raft_consensus.cc:2781] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:40.916894 31703 raft_consensus.cc:686] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 LEADER]: Becoming Leader. State: Replica: d6ea29dd0a4a427a9152c6193808eb27, State: Running, Role: LEADER
I20260927 22:15:40.917660 31703 consensus_queue.cc:237] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } }
I20260927 22:15:40.923856 30408 catalog_manager.cc:5495] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65). New cstate: current_term: 1 leader_uuid: "d6ea29dd0a4a427a9152c6193808eb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } health_report { overall_health: HEALTHY } } }
I20260927 22:15:40.988453 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:40.983369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.23.173.65:35279)
0927 22:15:40.983555 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:40.983559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:40.983615 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:15:40.983743 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:40.983751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:40.983876 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:40.983997 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:40.984343 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.985159 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.986467 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:40.986881 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:40.986973 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:40.987387 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:40.987908 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:40.988015 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:40.988093 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:15:40.988158 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:15:40.996971 31618 raft_consensus.cc:1261] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Refusing update from remote peer d6ea29dd0a4a427a9152c6193808eb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:40.996971 31477 raft_consensus.cc:1261] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Refusing update from remote peer d6ea29dd0a4a427a9152c6193808eb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:40.998092 31708 consensus_queue.cc:1035] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:40.998468 31703 consensus_queue.cc:1035] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:41.044479 30408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:41.044761 30408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:41.052816 30408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d298106e68b465aa6445ed4cdfb802f: Sending DeleteTablet for 3 replicas of tablet 02562d64a8a944f2a96e73ae9cee9057
I20260927 22:15:41.053879 31457 tablet_service.cc:1514] Processing DeleteTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:41 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:56720
I20260927 22:15:41.054044 31598 tablet_service.cc:1514] Processing DeleteTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:41 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:48786
I20260927 22:15:41.057248 31309 tablet_service.cc:1514] Processing DeleteTablet for tablet 02562d64a8a944f2a96e73ae9cee9057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:41 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:41726
I20260927 22:15:41.064641 31722 tablet_replica.cc:330] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: stopping tablet replica
I20260927 22:15:41.065094 31722 raft_consensus.cc:2229] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:41.065558 31722 raft_consensus.cc:2258] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:41.069069 31722 ts_tablet_manager.cc:1905] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:41.073726 31722 ts_tablet_manager.cc:1918] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:41.073895 31722 log.cc:1196] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/02562d64a8a944f2a96e73ae9cee9057
I20260927 22:15:41.074525 31722 ts_tablet_manager.cc:1939] T 02562d64a8a944f2a96e73ae9cee9057 P e9572de2364b423abdb0ce65deae31d5: Deleting consensus metadata
I20260927 22:15:41.075109 31721 tablet_replica.cc:330] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: stopping tablet replica
I20260927 22:15:41.075670 31721 raft_consensus.cc:2229] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:41.076221 31721 raft_consensus.cc:2258] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:41.076292 30396 catalog_manager.cc:4841] TS e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269): tablet 02562d64a8a944f2a96e73ae9cee9057 (table test-table [id=809661391e95420c92d554e7a552b8e5]) successfully deleted
I20260927 22:15:41.077724 31725 tablet_replica.cc:330] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: stopping tablet replica
I20260927 22:15:41.078153 31725 raft_consensus.cc:2229] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:41.078823 31725 raft_consensus.cc:2258] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:41.079999 31721 ts_tablet_manager.cc:1905] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:41.080168 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30377
I20260927 22:15:41.081168 30439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:41.088179 31721 ts_tablet_manager.cc:1918] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:41.089107 31725 ts_tablet_manager.cc:1905] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:41.096412 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:41.056603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.23.173.125:40493)
0927 22:15:41.062478 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:41.062488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:41.077122 (+ 14634us) server_negotiation.cc:407] Connection header received
0927 22:15:41.077269 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:41.077277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:41.077385 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:41.077501 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:41.081802 (+ 4301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.082974 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.084919 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.085566 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.085665 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:41.089295 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:41.089301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:41.089306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:41.089346 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:41.093134 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.094020 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.094026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.094030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.094206 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.094665 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.094670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.094674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.095026 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:41.095251 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:41.095799 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 22:15:41.096030 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5707,"thread_start_us":5581,"threads_started":1}
I20260927 22:15:41.096844 31721 log.cc:1196] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/02562d64a8a944f2a96e73ae9cee9057
I20260927 22:15:41.097514 31721 ts_tablet_manager.cc:1939] T 02562d64a8a944f2a96e73ae9cee9057 P d246fd58454b4b24bd6a9116051a931b: Deleting consensus metadata
I20260927 22:15:41.099679 31725 ts_tablet_manager.cc:1918] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:41.099920 31725 log.cc:1196] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/02562d64a8a944f2a96e73ae9cee9057
I20260927 22:15:41.101512 31725 ts_tablet_manager.cc:1939] T 02562d64a8a944f2a96e73ae9cee9057 P d6ea29dd0a4a427a9152c6193808eb27: Deleting consensus metadata
I20260927 22:15:41.112208 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:41.077260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.23.173.124:39759)
0927 22:15:41.080042 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:41.080048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:41.097882 (+ 17834us) server_negotiation.cc:407] Connection header received
0927 22:15:41.098193 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:41.098199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:41.098295 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:41.098376 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:41.100494 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.101654 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.103500 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.103934 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.104088 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:41.106737 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:41.106742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:41.106745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:41.106778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:41.109461 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.110216 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.110223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.110228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.110386 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.110746 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.110752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.110760 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.111014 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:41.111272 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:41.111603 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:15:41.111828 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2652,"thread_start_us":3298,"threads_started":1}
I20260927 22:15:41.268246 30393 catalog_manager.cc:4841] TS d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833): tablet 02562d64a8a944f2a96e73ae9cee9057 (table test-table [id=809661391e95420c92d554e7a552b8e5]) successfully deleted
I20260927 22:15:41.268286 30394 catalog_manager.cc:4841] TS d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279): tablet 02562d64a8a944f2a96e73ae9cee9057 (table test-table [id=809661391e95420c92d554e7a552b8e5]) successfully deleted
I20260927 22:15:41.268908 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30377
I20260927 22:15:41.312995 31726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:41.294498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.23.173.124:39759)
0927 22:15:41.294703 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:41.294710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:41.296182 (+ 1472us) server_negotiation.cc:407] Connection header received
0927 22:15:41.296472 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:41.296479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:41.296573 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:41.296650 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:41.299114 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.300218 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.302200 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.302896 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.302996 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:41.307083 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:41.307093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:41.307098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:41.307139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:41.310307 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.310931 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.310940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.310943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.311066 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.311464 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.311469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.311471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.311787 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:41.312034 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:41.312439 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 22:15:41.312647 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:15:41.320057 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:41.295444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.23.173.125:40493)
0927 22:15:41.295635 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:41.295641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:41.304997 (+ 9356us) server_negotiation.cc:407] Connection header received
0927 22:15:41.305198 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:41.305203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:41.305293 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:41.305406 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:41.307137 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.308265 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.310182 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:41.310854 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:41.310976 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:41.314330 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:41.314336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:41.314340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:41.314379 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:41.317491 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.318181 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.318186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.318190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.318334 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:41.318664 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:41.318669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:41.318673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:41.319006 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:41.319250 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:41.319491 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:15:41.319710 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:15:42.037108 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.036320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:37841)
0927 22:15:42.036787 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.036936 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
W20260927 22:15:42.037585 31664 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:42035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.043061 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.042297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:56479)
0927 22:15:42.042702 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.042866 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":211,"threads_started":1}
W20260927 22:15:42.043453 31375 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:42035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.047369 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.046663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:56715)
0927 22:15:42.047041 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.047150 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":178,"threads_started":1}
W20260927 22:15:42.047820 31523 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:42035 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.552893 31739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d298106e68b465aa6445ed4cdfb802f)
I20260927 22:15:42.552893 31738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d298106e68b465aa6445ed4cdfb802f)
I20260927 22:15:42.553190 31738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } }
I20260927 22:15:42.553193 31739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } }
I20260927 22:15:42.554839 31739 leader_election.cc:288] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035), 0f5210b4a5fd456f82f0e9c984db2cc7 (127.23.173.125:40493)
I20260927 22:15:42.555460 31738 leader_election.cc:288] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035), 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124:39759)
I20260927 22:15:42.556627 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.554946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45106)
0927 22:15:42.556114 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.556333 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1044,"threads_started":1}
W20260927 22:15:42.557045 30993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.557708 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.554944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45108)
0927 22:15:42.557235 (+ 2291us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.557412 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2164,"thread_start_us":2043,"threads_started":1}
W20260927 22:15:42.558728 31180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.560519 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.560043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45124)
0927 22:15:42.560222 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.560333 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:15:42.560690 30993 leader_election.cc:334] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035): Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.561009 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.560459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45134)
0927 22:15:42.560634 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.560810 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 22:15:42.561275 31180 leader_election.cc:334] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035): Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.564606 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.555458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.23.173.124:39759)
0927 22:15:42.556378 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.556383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.556922 (+ 539us) server_negotiation.cc:407] Connection header received
0927 22:15:42.557206 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.557213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.557304 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.557389 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:42.558329 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.559400 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.562747 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.563760 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.563872 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.563920 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.564238 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.564334 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":809,"thread_start_us":660,"threads_started":1}
I20260927 22:15:42.564760 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.555040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39759 (local address 127.0.0.1:42218)
0927 22:15:42.556824 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.556842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:42.557040 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:42.557668 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:42.557673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:42.557706 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:42.558091 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.558105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.559575 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.559580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.562585 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.562610 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.563904 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.563910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.564083 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.564122 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:15:42.564220 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:15:42.564348 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1668,"thread_start_us":828,"threads_started":1}
I20260927 22:15:42.565392 31205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3d016a30e663494ba8c021e51bf3b650" is_pre_election: true
I20260927 22:15:42.565747 31205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f5210b4a5fd456f82f0e9c984db2cc7 in term 1.
I20260927 22:15:42.566246 30992 leader_election.cc:302] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f5210b4a5fd456f82f0e9c984db2cc7, 3d016a30e663494ba8c021e51bf3b650; no voters: 9d298106e68b465aa6445ed4cdfb802f
I20260927 22:15:42.566707 31738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:15:42.566834 31738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d298106e68b465aa6445ed4cdfb802f)
I20260927 22:15:42.566955 31738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:15:42.567170 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.557356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:40493 (local address 127.0.0.1:41170)
0927 22:15:42.557985 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.558003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:42.558111 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:42.561164 (+ 3053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:42.561169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:42.561197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:42.561515 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.561527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.562844 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.562848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.564717 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.564733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.566347 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.566352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.566529 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.566573 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:15:42.566709 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:15:42.566854 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1,"thread_start_us":996,"threads_started":1}
I20260927 22:15:42.567306 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.557666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.23.173.125:40493)
0927 22:15:42.560840 (+ 3174us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.560847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.560874 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:15:42.560927 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.560931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.561000 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.561102 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:42.561675 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.562664 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.564908 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.566148 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.566336 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.566412 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.566740 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.566870 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3030,"thread_start_us":2844,"threads_started":1}
I20260927 22:15:42.567865 31017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d016a30e663494ba8c021e51bf3b650" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" is_pre_election: true
I20260927 22:15:42.569552 31738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } }
I20260927 22:15:42.569940 31017 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d016a30e663494ba8c021e51bf3b650 in current term 2: Already voted for candidate 0f5210b4a5fd456f82f0e9c984db2cc7 in this term.
I20260927 22:15:42.570207 31738 leader_election.cc:288] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 election: Requested vote from peers 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035), 3d016a30e663494ba8c021e51bf3b650 (127.23.173.124:39759)
I20260927 22:15:42.570694 31205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3d016a30e663494ba8c021e51bf3b650"
I20260927 22:15:42.570926 31205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:15:42.571026 31180 leader_election.cc:302] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d016a30e663494ba8c021e51bf3b650; no voters: 0f5210b4a5fd456f82f0e9c984db2cc7, 9d298106e68b465aa6445ed4cdfb802f
I20260927 22:15:42.571274 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.570868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45144)
0927 22:15:42.571010 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.571110 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 22:15:42.572273 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.571915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45146)
0927 22:15:42.572054 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.572138 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:15:42.572455 30993 leader_election.cc:334] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035): Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
I20260927 22:15:42.573491 31205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f5210b4a5fd456f82f0e9c984db2cc7 in term 2.
I20260927 22:15:42.573680 31739 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:15:42.573912 30992 leader_election.cc:302] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f5210b4a5fd456f82f0e9c984db2cc7, 3d016a30e663494ba8c021e51bf3b650; no voters: 9d298106e68b465aa6445ed4cdfb802f
I20260927 22:15:42.574234 31738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:15:42.574638 31738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [term 2 LEADER]: Becoming Leader. State: Replica: 0f5210b4a5fd456f82f0e9c984db2cc7, State: Running, Role: LEADER
I20260927 22:15:42.574987 31738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } }
I20260927 22:15:42.576736 31748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f5210b4a5fd456f82f0e9c984db2cc7. Latest consensus state: current_term: 2 leader_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:42.577131 31748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:42.577734 31750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:42.581103 31750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:42.581900 31750 catalog_manager.cc:1259] Loaded cluster ID: 67f3e39cfa0641cc89d68dba2ca580b0
I20260927 22:15:42.582031 31750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:42.583139 31750 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:42.583923 31750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Loaded TSK: 0
I20260927 22:15:42.584712 31750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:15:42.606976 31007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:42.613790 31007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:42.617615 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.617243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45166)
0927 22:15:42.617416 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.617487 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:15:42.617795 31205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [term 2 FOLLOWER]: Refusing update from remote peer 0f5210b4a5fd456f82f0e9c984db2cc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 22:15:42.618531 31748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 22:15:42.618706 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.618285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45182)
0927 22:15:42.618463 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.618559 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:15:42.618911 30993 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 -> Peer 9d298106e68b465aa6445ed4cdfb802f (127.23.173.126:42035): Couldn't send request to peer 9d298106e68b465aa6445ed4cdfb802f. Status: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:15:42.621021 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f5210b4a5fd456f82f0e9c984db2cc7. Latest consensus state: current_term: 2 leader_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:42.621292 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:42.623112 31752 mvcc.cc:205] Tried to move back new op lower bound from 7334081915355914240 to 7334081915190341632. Current Snapshot: MvccSnapshot[applied={T|T < 7334081915355914240}]
I20260927 22:15:42.623039 31738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:42.623394 31738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:42.624200 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d298106e68b465aa6445ed4cdfb802f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42035 } } peers { permanent_uuid: "0f5210b4a5fd456f82f0e9c984db2cc7" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 40493 } attrs { promote: false } } peers { permanent_uuid: "3d016a30e663494ba8c021e51bf3b650" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39759 } attrs { promote: false } } }
I20260927 22:15:42.624488 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d016a30e663494ba8c021e51bf3b650 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:15:42.641593 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.631858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:44833 (local address 127.0.0.1:56736)
0927 22:15:42.632128 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.632152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:42.632252 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:42.635578 (+ 3326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:42.635583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:42.635608 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:42.635895 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.635903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.636944 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.636952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.639199 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.639213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.640786 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.640792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.641126 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.641174 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:15:42.641260 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:15:42.641372 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":76}
I20260927 22:15:42.642307 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.632057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.23.173.66:44833)
0927 22:15:42.635249 (+ 3192us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.635255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.635274 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:15:42.635349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.635354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.635428 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.636003 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:42.636115 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.636819 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.639337 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.640556 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.640687 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.640802 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.641919 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.642096 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3077,"thread_start_us":3478,"threads_started":1}
I20260927 22:15:42.643011 31457 tablet_service.cc:1467] Processing CreateTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 (DEFAULT_TABLE table=test-table [id=e496ea84218b431dae5e069ef699ed30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:42.643151 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.630918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:35279 (local address 127.0.0.1:41738)
0927 22:15:42.631067 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.631085 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:42.631210 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:42.637987 (+ 6777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:42.637994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:42.638029 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:42.638218 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.638228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.639244 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.639251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.641120 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.641134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.642635 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.642642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.642801 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.642824 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:15:42.642892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:15:42.642978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":95}
I20260927 22:15:42.643489 31457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bfaf8c50a340b9a4b76540e654a283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:42.644589 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.631934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:37269 (local address 127.0.0.1:48796)
0927 22:15:42.637475 (+ 5541us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.637502 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:42.637606 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:42.638371 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:42.638375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:42.638395 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:42.638780 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.638792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.640192 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.640198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.642335 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:42.642350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.643871 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.643878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:42.644041 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.644071 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:15:42.644173 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:15:42.644254 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5411,"mutex_wait_us":44,"thread_start_us":5174,"threads_started":1}
I20260927 22:15:42.645217 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.632351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.23.173.67:37269)
0927 22:15:42.634354 (+ 2003us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.634363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.637569 (+ 3206us) server_negotiation.cc:407] Connection header received
0927 22:15:42.637754 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.637759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.637830 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.637938 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:42.638953 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.640014 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.642524 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.643655 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.643782 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.643876 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.644803 (+ 927us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.644918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1853,"thread_start_us":1767,"threads_started":1}
I20260927 22:15:42.645888 31598 tablet_service.cc:1467] Processing CreateTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 (DEFAULT_TABLE table=test-table [id=e496ea84218b431dae5e069ef699ed30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:42.646360 31598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bfaf8c50a340b9a4b76540e654a283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:42.646932 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.631022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.23.173.65:35279)
0927 22:15:42.637653 (+ 6631us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.637659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.637688 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:15:42.637752 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.637756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.637841 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.637938 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:42.638351 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.639108 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.641287 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.642506 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.642627 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.646536 (+ 3909us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.646579 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.646715 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6515,"thread_start_us":6418,"threads_started":1}
I20260927 22:15:42.650481 31309 tablet_service.cc:1467] Processing CreateTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 (DEFAULT_TABLE table=test-table [id=e496ea84218b431dae5e069ef699ed30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:42.650873 31309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bfaf8c50a340b9a4b76540e654a283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:42.656041 31761 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Bootstrap starting.
I20260927 22:15:42.657133 31763 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Bootstrap starting.
I20260927 22:15:42.657929 31761 tablet_bootstrap.cc:654] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:42.658567 31762 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Bootstrap starting.
I20260927 22:15:42.659189 31763 tablet_bootstrap.cc:654] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:42.660292 31761 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: No bootstrap required, opened a new log
I20260927 22:15:42.660332 31762 tablet_bootstrap.cc:654] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:42.660440 31761 ts_tablet_manager.cc:1397] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:15:42.661216 31763 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: No bootstrap required, opened a new log
I20260927 22:15:42.661450 31763 ts_tablet_manager.cc:1397] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 22:15:42.661450 31761 raft_consensus.cc:348] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.661746 31761 raft_consensus.cc:374] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:42.661860 31761 raft_consensus.cc:729] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d246fd58454b4b24bd6a9116051a931b, State: Initialized, Role: FOLLOWER
I20260927 22:15:42.662169 31761 consensus_queue.cc:260] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.662644 31762 tablet_bootstrap.cc:492] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: No bootstrap required, opened a new log
I20260927 22:15:42.662796 31762 ts_tablet_manager.cc:1397] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:15:42.662690 31763 raft_consensus.cc:348] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.662992 31763 raft_consensus.cc:374] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:42.663092 31763 raft_consensus.cc:729] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ea29dd0a4a427a9152c6193808eb27, State: Initialized, Role: FOLLOWER
I20260927 22:15:42.663329 31763 consensus_queue.cc:260] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.663580 31762 raft_consensus.cc:348] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.663854 31762 raft_consensus.cc:374] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:42.663955 31762 raft_consensus.cc:729] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9572de2364b423abdb0ce65deae31d5, State: Initialized, Role: FOLLOWER
I20260927 22:15:42.664227 31763 ts_tablet_manager.cc:1428] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:42.664219 31762 consensus_queue.cc:260] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.672508 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.664889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:47823)
0927 22:15:42.671940 (+ 7051us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.672167 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6880,"thread_start_us":6695,"threads_started":1}
I20260927 22:15:42.673295 31761 ts_tablet_manager.cc:1428] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260927 22:15:42.675498 31762 ts_tablet_manager.cc:1428] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Time spent starting tablet: real 0.013s user 0.000s sys 0.011s
I20260927 22:15:42.678504 31376 heartbeater.cc:503] Master 127.23.173.125:40493 was elected leader, sending a full tablet report...
I20260927 22:15:42.679847 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.679318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:36263)
0927 22:15:42.679535 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.679656 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:15:42.680598 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.674750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:36257)
0927 22:15:42.680158 (+ 5408us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.680352 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5021,"threads_started":1}
I20260927 22:15:42.681697 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.675260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:46209)
0927 22:15:42.681254 (+ 5994us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.681422 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5850,"thread_start_us":5561,"threads_started":1}
I20260927 22:15:42.682688 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.682275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:59077)
0927 22:15:42.682469 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.682554 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 22:15:42.682842 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.682344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:59145)
0927 22:15:42.682520 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.682586 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:15:42.683204 31665 heartbeater.cc:503] Master 127.23.173.125:40493 was elected leader, sending a full tablet report...
I20260927 22:15:42.683230 31524 heartbeater.cc:503] Master 127.23.173.125:40493 was elected leader, sending a full tablet report...
I20260927 22:15:42.844059 31768 raft_consensus.cc:482] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:42.844303 31768 raft_consensus.cc:504] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.845062 31768 leader_election.cc:288] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269), d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
I20260927 22:15:42.845712 31618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9bfaf8c50a340b9a4b76540e654a283" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9572de2364b423abdb0ce65deae31d5" is_pre_election: true
I20260927 22:15:42.845723 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9bfaf8c50a340b9a4b76540e654a283" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d246fd58454b4b24bd6a9116051a931b" is_pre_election: true
I20260927 22:15:42.845948 31618 raft_consensus.cc:2454] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 0.
I20260927 22:15:42.846029 31477 raft_consensus.cc:2454] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 0.
I20260927 22:15:42.846508 31265 leader_election.cc:302] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ea29dd0a4a427a9152c6193808eb27, e9572de2364b423abdb0ce65deae31d5; no voters:
I20260927 22:15:42.846781 31768 raft_consensus.cc:2781] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:42.846920 31768 raft_consensus.cc:482] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:42.847010 31768 raft_consensus.cc:3037] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:42.849304 31768 raft_consensus.cc:504] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.849864 31768 leader_election.cc:288] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 election: Requested vote from peers e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269), d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833)
I20260927 22:15:42.850406 31618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9bfaf8c50a340b9a4b76540e654a283" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9572de2364b423abdb0ce65deae31d5"
I20260927 22:15:42.850468 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9bfaf8c50a340b9a4b76540e654a283" candidate_uuid: "d6ea29dd0a4a427a9152c6193808eb27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d246fd58454b4b24bd6a9116051a931b"
I20260927 22:15:42.850608 31618 raft_consensus.cc:3037] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:42.850675 31477 raft_consensus.cc:3037] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:42.852732 31618 raft_consensus.cc:2454] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 1.
I20260927 22:15:42.852737 31477 raft_consensus.cc:2454] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ea29dd0a4a427a9152c6193808eb27 in term 1.
I20260927 22:15:42.853174 31262 leader_election.cc:302] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d246fd58454b4b24bd6a9116051a931b, d6ea29dd0a4a427a9152c6193808eb27; no voters:
I20260927 22:15:42.853462 31768 raft_consensus.cc:2781] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:42.853888 31768 raft_consensus.cc:686] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 LEADER]: Becoming Leader. State: Replica: d6ea29dd0a4a427a9152c6193808eb27, State: Running, Role: LEADER
I20260927 22:15:42.854254 31768 consensus_queue.cc:237] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } }
I20260927 22:15:42.855177 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.854686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:41157)
0927 22:15:42.854941 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.855033 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 22:15:42.857191 31008 catalog_manager.cc:5495] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65). New cstate: current_term: 1 leader_uuid: "d6ea29dd0a4a427a9152c6193808eb27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ea29dd0a4a427a9152c6193808eb27" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 35279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:42.877672 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.873426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.23.173.65:35279)
0927 22:15:42.873658 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.873662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.873680 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:42.873838 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.873842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.873917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.874052 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:42.874359 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.875023 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.876204 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.876614 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.876704 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.877026 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:42.877120 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:42.877195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.877313 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.877374 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:15:42.880689 31477 raft_consensus.cc:1261] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Refusing update from remote peer d6ea29dd0a4a427a9152c6193808eb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:42.880703 31618 raft_consensus.cc:1261] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Refusing update from remote peer d6ea29dd0a4a427a9152c6193808eb27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:42.881335 31773 consensus_queue.cc:1035] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d246fd58454b4b24bd6a9116051a931b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:42.881661 31768 consensus_queue.cc:1035] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9572de2364b423abdb0ce65deae31d5" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:42.884215 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.883520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:54679)
0927 22:15:42.883811 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.883954 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 22:15:42.884265 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.883631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:58833)
0927 22:15:42.883872 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.883989 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 22:15:42.888244 31777 mvcc.cc:205] Tried to move back new op lower bound from 7334081916433268736 to 7334081916333854720. Current Snapshot: MvccSnapshot[applied={T|T < 7334081916433268736}]
I20260927 22:15:42.889822 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.889273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.66:42541)
0927 22:15:42.889590 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.889677 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 22:15:42.891013 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.890378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:54881)
0927 22:15:42.890764 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.890876 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
I20260927 22:15:42.894649 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.894180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.65:54865)
0927 22:15:42.894375 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.894456 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 22:15:42.894753 31778 mvcc.cc:205] Tried to move back new op lower bound from 7334081916433268736 to 7334081916333854720. Current Snapshot: MvccSnapshot[applied={T|T < 7334081916433268736}]
I20260927 22:15:42.896950 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.896463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.23.173.67:48489)
0927 22:15:42.896700 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:42.896793 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 22:15:42.901582 31008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:42.901806 31008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:42.906736 31008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f5210b4a5fd456f82f0e9c984db2cc7: Sending DeleteTablet for 3 replicas of tablet d9bfaf8c50a340b9a4b76540e654a283
I20260927 22:15:42.907680 31309 tablet_service.cc:1514] Processing DeleteTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:42 UTC) from {username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:41738
I20260927 22:15:42.907943 31598 tablet_service.cc:1514] Processing DeleteTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:42 UTC) from {username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:48796
I20260927 22:15:42.908082 31457 tablet_service.cc:1514] Processing DeleteTablet for tablet d9bfaf8c50a340b9a4b76540e654a283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:42 UTC) from {username='oryx', principal='oryx/127.23.173.125@KRBTEST.COM'} at 127.0.0.1:56736
I20260927 22:15:42.908535 31785 tablet_replica.cc:330] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: stopping tablet replica
I20260927 22:15:42.908730 31787 tablet_replica.cc:330] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: stopping tablet replica
I20260927 22:15:42.908772 31785 raft_consensus.cc:2229] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:42.908921 31787 raft_consensus.cc:2229] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:42.909216 31787 raft_consensus.cc:2258] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:42.909364 31785 raft_consensus.cc:2258] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:42.910041 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 31232
I20260927 22:15:42.910434 31787 ts_tablet_manager.cc:1905] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:42.910460 31785 ts_tablet_manager.cc:1905] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:42.916749 31787 ts_tablet_manager.cc:1918] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:42.916749 31785 ts_tablet_manager.cc:1918] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:42.916877 31787 log.cc:1196] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/d9bfaf8c50a340b9a4b76540e654a283
I20260927 22:15:42.917014 31785 log.cc:1196] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/d9bfaf8c50a340b9a4b76540e654a283
I20260927 22:15:42.917317 31787 ts_tablet_manager.cc:1939] T d9bfaf8c50a340b9a4b76540e654a283 P d246fd58454b4b24bd6a9116051a931b: Deleting consensus metadata
I20260927 22:15:42.917577 31785 ts_tablet_manager.cc:1939] T d9bfaf8c50a340b9a4b76540e654a283 P d6ea29dd0a4a427a9152c6193808eb27: Deleting consensus metadata
I20260927 22:15:42.918594 30993 catalog_manager.cc:4841] TS d6ea29dd0a4a427a9152c6193808eb27 (127.23.173.65:35279): tablet d9bfaf8c50a340b9a4b76540e654a283 (table test-table [id=e496ea84218b431dae5e069ef699ed30]) successfully deleted
I20260927 22:15:42.918545 30992 catalog_manager.cc:4841] TS d246fd58454b4b24bd6a9116051a931b (127.23.173.66:44833): tablet d9bfaf8c50a340b9a4b76540e654a283 (table test-table [id=e496ea84218b431dae5e069ef699ed30]) successfully deleted
I20260927 22:15:42.922868 31786 tablet_replica.cc:330] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: stopping tablet replica
I20260927 22:15:42.923061 31786 raft_consensus.cc:2229] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:42.923336 31786 raft_consensus.cc:2258] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:42.924292 31786 ts_tablet_manager.cc:1905] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:42.928517 31786 ts_tablet_manager.cc:1918] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:42.928663 31786 log.cc:1196] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/d9bfaf8c50a340b9a4b76540e654a283
I20260927 22:15:42.929018 31786 ts_tablet_manager.cc:1939] T d9bfaf8c50a340b9a4b76540e654a283 P e9572de2364b423abdb0ce65deae31d5: Deleting consensus metadata
I20260927 22:15:42.929877 30995 catalog_manager.cc:4841] TS e9572de2364b423abdb0ce65deae31d5 (127.23.173.67:37269): tablet d9bfaf8c50a340b9a4b76540e654a283 (table test-table [id=e496ea84218b431dae5e069ef699ed30]) successfully deleted
I20260927 22:15:42.935321 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:42.912084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.23.173.65:35279)
0927 22:15:42.912298 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:42.912312 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:42.922427 (+ 10115us) server_negotiation.cc:407] Connection header received
0927 22:15:42.922560 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:42.922569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:42.922691 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:42.922792 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:42.924260 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.925029 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.926280 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:42.926692 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:42.926830 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:42.929689 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:42.929709 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:42.929732 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:42.929774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:42.932599 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:42.933245 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:42.933255 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:42.933258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:42.933395 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:42.933781 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:42.933785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:42.933789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:42.934209 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:42.934503 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:42.934794 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:15:42.935013 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:15:42.935945 31369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:43.022238 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:43.021397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45186)
0927 22:15:43.021771 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:43.021936 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 22:15:43.023417 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:43.022945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42035 (local address 127.0.0.1:45202)
0927 22:15:43.023128 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:43.023251 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:15:43.059412 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 31232
I20260927 22:15:43.081940 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 31386
I20260927 22:15:43.095659 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:43.082820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.23.173.66:44833)
0927 22:15:43.083120 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:43.083126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:43.083148 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:15:43.083330 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:43.083334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:43.083397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:43.083506 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:43.084858 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:43.085552 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:43.087002 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:43.087385 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:43.087555 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:43.090191 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:43.090204 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:43.090220 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:43.090253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:43.092940 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:43.093598 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:43.093614 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:43.093617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:43.093738 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:43.094186 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:43.094192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:43.094195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:43.094527 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:43.094828 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:43.095141 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:15:43.095372 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:15:43.096295 31517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:43.189694 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 31386
I20260927 22:15:43.207042 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 31535
I20260927 22:15:43.220532 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:43.207841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.23.173.67:37269)
0927 22:15:43.208329 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:43.208334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:43.208350 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:15:43.208404 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:43.208407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:43.208471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:43.208591 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:43.209932 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:43.210706 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:43.211904 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:43.212340 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:43.212477 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:43.215069 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:43.215082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:43.215098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:43.215129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:43.217805 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:43.218515 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:43.218534 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:43.218537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:43.218654 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:43.219026 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:43.219033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:43.219036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:43.219398 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:43.219740 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:43.220000 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 22:15:43.220219 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260927 22:15:43.221186 31658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:43.313983 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 31535
I20260927 22:15:43.331300 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30870
I20260927 22:15:43.332362 31038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:43.456029 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30870
I20260927 22:15:43.477566 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 30971
I20260927 22:15:43.478721 31225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:43.595300 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 30971
2026-09-27T22:15:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9748 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:43 dist-test-slave-3wc3 krb5kdc[31797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:43 dist-test-slave-3wc3 krb5kdc[31797](info): set up 2 sockets
Sep 27 22:15:43 dist-test-slave-3wc3 krb5kdc[31797](info): commencing operation
krb5kdc: starting...
W20260927 22:15:44.698944 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.012s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:44 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab.
Entry for principal oryx/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab.
I20260927 22:15:44.917857 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38437
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab
--principal=oryx/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:45.242602 31810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:45.242906 31810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:45.243067 31810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:45.250227 31810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:45.250365 31810 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:15:45.250442 31810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:45.250552 31810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:45.250643 31810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:45.250756 31810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:45.260406 31810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:38437
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.126.keytab
--principal=oryx/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:45.261650 31810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:45.263013 31810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:45.274912 31816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:45.274914 31815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:45.274995 31818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:45.275686 31810 server_base.cc:1034] running on GCE node
I20260927 22:15:45.276562 31810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:45.278045 31810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:45.279281 31810 hybrid_clock.cc:648] HybridClock initialized: now 1790547345279211 us; error 51 us; skew 500 ppm
I20260927 22:15:45.279605 31810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:45 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547345, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:45.282756 31810 init.cc:377] Logged in from keytab as oryx/127.23.173.126@KRBTEST.COM (short username oryx)
I20260927 22:15:45.284159 31810 webserver.cc:460] Webserver started at http://127.23.173.126:36107/ using document root <none> and password file <none>
I20260927 22:15:45.284868 31810 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:45.284996 31810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:45.285315 31810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:45.288359 31810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "f9401860d4bc48d49f515f34658357e0"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "92e348cd8e32063b28e0358a4c10943a"
server_key_iv: "9a9cc361eda7ec1b2dca7fd8f5866f55"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.289165 31810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "f9401860d4bc48d49f515f34658357e0"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "92e348cd8e32063b28e0358a4c10943a"
server_key_iv: "9a9cc361eda7ec1b2dca7fd8f5866f55"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.294210 31810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:15:45.297571 31825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:45.298923 31810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:45.299106 31810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "f9401860d4bc48d49f515f34658357e0"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "92e348cd8e32063b28e0358a4c10943a"
server_key_iv: "9a9cc361eda7ec1b2dca7fd8f5866f55"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.299327 31810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:45.317291 31810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:45.321674 31810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:45.322008 31810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:45.343556 31810 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:38437
I20260927 22:15:45.343576 31886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:38437 every 8 connection(s)
I20260927 22:15:45.345001 31810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:45.348035 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 31810
I20260927 22:15:45.348244 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:45.349107 24245 external_mini_cluster.cc:1428] Setting key b8c962e7a4182c1102ca1fa0663abe10
I20260927 22:15:45.349249 31887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:45.363938 31887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Bootstrap starting.
Sep 27 22:15:45 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:45.367774 31887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:45.368894 31887 log.cc:824] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:45.372125 31887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: No bootstrap required, opened a new log
I20260927 22:15:45.372923 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:45.351239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.23.173.126:38437)
0927 22:15:45.351958 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:45.351975 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:45.352033 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:15:45.354436 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:45.354466 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:45.354986 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:45.355439 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:45.356734 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.358754 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.360083 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.361194 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.361386 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:45.365582 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:45.365590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:45.365603 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:45.365649 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:45.368725 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.369553 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.369565 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.369576 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.369813 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.370242 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.370247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.370250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.370783 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:45.370981 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:45.371475 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 22:15:45.371860 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":293,"threads_started":1}
I20260927 22:15:45.379628 31887 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } }
I20260927 22:15:45.380023 31887 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:45.380134 31887 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9401860d4bc48d49f515f34658357e0, State: Initialized, Role: FOLLOWER
I20260927 22:15:45.380749 31887 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } }
I20260927 22:15:45.380980 31887 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:45.381114 31887 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:45.381282 31887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:45.383514 31887 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } }
I20260927 22:15:45.384004 31887 leader_election.cc:302] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9401860d4bc48d49f515f34658357e0; no voters:
I20260927 22:15:45.384584 31887 leader_election.cc:288] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:45.384747 31892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:45.386026 31892 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [term 1 LEADER]: Becoming Leader. State: Replica: f9401860d4bc48d49f515f34658357e0, State: Running, Role: LEADER
I20260927 22:15:45.386693 31887 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:45.386711 31892 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } }
I20260927 22:15:45.389881 31893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9401860d4bc48d49f515f34658357e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } } }
I20260927 22:15:45.389971 31894 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9401860d4bc48d49f515f34658357e0. Latest consensus state: current_term: 1 leader_uuid: "f9401860d4bc48d49f515f34658357e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9401860d4bc48d49f515f34658357e0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38437 } } }
I20260927 22:15:45.390295 31893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:45.390326 31894 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:45.397987 31899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:45.402498 31899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:45.410826 31899 catalog_manager.cc:1347] Generated new cluster ID: dae2e8244eac41c18665c6180a69225f
I20260927 22:15:45.410936 31899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:45.420888 31899 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:45.422554 31899 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:45.438431 31899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Generated new TSK 0
I20260927 22:15:45.439339 31899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
I20260927 22:15:45.499742 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38437
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:45.839242 31913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:45.839548 31913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:45.839730 31913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:45.847007 31913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:45.847148 31913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:45.847302 31913 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:15:45.847440 31913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:45.857214 31913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:45.858626 31913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:45.859964 31913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:45.873365 31919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:45.873453 31921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:45.873884 31918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:45.874248 31913 server_base.cc:1034] running on GCE node
I20260927 22:15:45.874934 31913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:45.875871 31913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:45.877084 31913 hybrid_clock.cc:648] HybridClock initialized: now 1790547345877002 us; error 64 us; skew 500 ppm
I20260927 22:15:45.877393 31913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:45 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547345, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:45.880570 31913 init.cc:377] Logged in from keytab as oryx/127.23.173.65@KRBTEST.COM (short username oryx)
I20260927 22:15:45.882154 31913 webserver.cc:460] Webserver started at http://127.23.173.65:43813/ using document root <none> and password file <none>
I20260927 22:15:45.882872 31913 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:45.883002 31913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:45.883319 31913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:45.886291 31913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "acc83231bddf4f5a867ade4270bb2f5c"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "82e15094b97d8e7856460c77ecb356e8"
server_key_iv: "a37be1b56582b66f819d810e489a9f01"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.887117 31913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "acc83231bddf4f5a867ade4270bb2f5c"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "82e15094b97d8e7856460c77ecb356e8"
server_key_iv: "a37be1b56582b66f819d810e489a9f01"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.892134 31913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:15:45.895605 31928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:45.896968 31913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:15:45.897095 31913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "acc83231bddf4f5a867ade4270bb2f5c"
format_stamp: "Formatted at 2026-09-27 22:15:45 on dist-test-slave-3wc3"
server_key: "82e15094b97d8e7856460c77ecb356e8"
server_key_iv: "a37be1b56582b66f819d810e489a9f01"
server_key_version: "encryptionkey@0"
I20260927 22:15:45.897264 31913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:45.916865 31913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:45.920516 31913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:45.920733 31913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:45.923255 31913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:45.923339 31913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:45.923496 31913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:45.923590 31913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:15:45 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547345, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:45.946751 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:45.923738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:36191 (local address 127.23.173.126:38437)
0927 22:15:45.926524 (+ 2786us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:45.926528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:45.926547 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:15:45.928889 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:45.928897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:45.929001 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:45.929144 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:45.931755 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.932715 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.935292 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.935724 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.935858 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:45.939757 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:45.939762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:45.939766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:45.939798 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:45.942636 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.943724 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.943731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.943735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.943898 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.944427 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.944433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.944436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.944832 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:45.945009 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:45.946148 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0927 22:15:45.946394 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2561,"threads_started":1}
I20260927 22:15:45.947633 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:45.924500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.65:36191)
0927 22:15:45.925575 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:45.925631 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:45.928528 (+ 2897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:45.929424 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:45.929439 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:45.930391 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:45.931480 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:45.931510 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.932969 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.932980 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:45.935127 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:45.935142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.935966 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.935974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:45.936201 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:45.936899 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:45.936919 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:45.939510 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:45.942796 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:45.942804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:45.942824 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:45.943568 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:45.944049 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:45.944057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:45.944065 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:45.944299 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:45.944991 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:45.944999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:45.945496 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 22:15:45.945865 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 22:15:45.946414 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":417,"threads_started":1}
I20260927 22:15:45.968511 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:45.953683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:35165 (local address 127.23.173.126:38437)
0927 22:15:45.953830 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:45.953834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:45.954740 (+ 906us) server_negotiation.cc:407] Connection header received
0927 22:15:45.954990 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:45.954996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:45.955095 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:45.955166 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:45.956697 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.957729 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.959790 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.960191 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.960328 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:45.962202 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:45.962207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:45.962210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:45.962239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:45.965349 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.966139 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.966153 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.966157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.966346 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:45.966771 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:45.966778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:45.966786 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:45.967179 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:45.967335 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:45.967896 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 22:15:45.968128 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:15:45.968583 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:45.953731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.65:35165)
0927 22:15:45.954621 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:45.954649 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:45.954812 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:45.955341 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:45.955344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:45.956026 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:45.956521 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:45.956538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.957861 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.957866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:45.959634 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:45.959646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:45.960375 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:45.960381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:45.960568 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:45.961229 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:45.961245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:45.961891 (+ 646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:45.965487 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:45.965492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:45.965496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:45.965995 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:45.966482 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:45.966489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:45.966493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:45.966641 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:45.967299 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:45.967308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:45.967553 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 22:15:45.967773 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 22:15:45.968140 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":494,"threads_started":1}
I20260927 22:15:45.980907 31913 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:34367
I20260927 22:15:45.980929 32048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:34367 every 8 connection(s)
I20260927 22:15:45.982200 31913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:45.984292 32049 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38437
I20260927 22:15:45.984328 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 31913
I20260927 22:15:45.984503 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:45.984514 32049 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:45.985119 24245 external_mini_cluster.cc:1428] Setting key a8cb7abe9357a4527c6c265dc6997cc2
I20260927 22:15:45.985159 32049 heartbeater.cc:511] Master 127.23.173.126:38437 requested a full tablet report, sending...
I20260927 22:15:45.987388 31841 ts_manager.cc:194] Registered new tserver with Master: acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367)
I20260927 22:15:45.989552 31841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:36191
WARNING: no policy specified for oryx/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
I20260927 22:15:46.031424 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38437
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:46.356139 32053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:46.356407 32053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:46.356570 32053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:46.363793 32053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:46.363893 32053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:46.364055 32053 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:15:46.364194 32053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:46.373899 32053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:46.375198 32053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:46.376549 32053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:46.389786 32058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:46.389868 32061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:46.389886 32059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:46.390599 32053 server_base.cc:1034] running on GCE node
I20260927 22:15:46.391222 32053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:46.392093 32053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:46.393338 32053 hybrid_clock.cc:648] HybridClock initialized: now 1790547346393259 us; error 58 us; skew 500 ppm
I20260927 22:15:46.393646 32053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:46 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547346, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:46.397038 32053 init.cc:377] Logged in from keytab as oryx/127.23.173.66@KRBTEST.COM (short username oryx)
I20260927 22:15:46.398578 32053 webserver.cc:460] Webserver started at http://127.23.173.66:40883/ using document root <none> and password file <none>
I20260927 22:15:46.399327 32053 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:46.399453 32053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:46.399847 32053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:46.403038 32053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "8048118b5cee4c2ca00975cc19f7db40"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "ad77b4d206c49f17768d19082fb7d1e5"
server_key_iv: "b2f9187a3526ac87423c6bde3ae8e02c"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.403973 32053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "8048118b5cee4c2ca00975cc19f7db40"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "ad77b4d206c49f17768d19082fb7d1e5"
server_key_iv: "b2f9187a3526ac87423c6bde3ae8e02c"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.409216 32053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:15:46.412777 32068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:46.414235 32053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:46.414404 32053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "8048118b5cee4c2ca00975cc19f7db40"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "ad77b4d206c49f17768d19082fb7d1e5"
server_key_iv: "b2f9187a3526ac87423c6bde3ae8e02c"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.414635 32053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:46.427137 32053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:46.431144 32053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:46.431443 32053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:46.434109 32053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:46.434190 32053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:46.434351 32053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:46.434423 32053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:15:46 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547346, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:46.459331 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.434966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:59453 (local address 127.23.173.126:38437)
0927 22:15:46.435167 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:46.435173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:46.436555 (+ 1382us) server_negotiation.cc:407] Connection header received
0927 22:15:46.439052 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:46.439058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:46.439149 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:46.439271 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:46.441402 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.442755 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.445091 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.445735 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.445869 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.451528 (+ 5659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:46.451535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:46.451539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:46.451585 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:46.455741 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.456816 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.456823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.456827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.456965 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.457420 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.457426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.457429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.457744 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:46.457915 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:46.458726 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 22:15:46.458953 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:15:46.460933 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.435487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.66:59453)
0927 22:15:46.436360 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:46.436404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:46.438691 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:46.439568 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:46.439585 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:46.440320 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:46.441155 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.441174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.442946 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.442953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.444922 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.444934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.446017 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.446025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.446345 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.447303 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:46.447347 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:46.451277 (+ 3930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:46.455945 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.455953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.455967 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.456667 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.457085 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.457090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.457093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.457300 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.457883 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:46.457891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:46.458362 (+ 471us) client_negotiation.cc:769] Sending connection context
0927 22:15:46.458651 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 22:15:46.459266 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":390,"threads_started":1}
I20260927 22:15:46.481420 32053 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:46555
I20260927 22:15:46.481446 32187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:46555 every 8 connection(s)
I20260927 22:15:46.482923 32053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:46.485272 32188 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38437
I20260927 22:15:46.485543 32188 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:46.486295 32188 heartbeater.cc:511] Master 127.23.173.126:38437 requested a full tablet report, sending...
I20260927 22:15:46.487568 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.468656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:35217 (local address 127.23.173.126:38437)
0927 22:15:46.468912 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:46.468921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:46.471519 (+ 2598us) server_negotiation.cc:407] Connection header received
0927 22:15:46.471774 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:46.471779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:46.471846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:46.471934 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:46.473543 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.474447 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.477424 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.477991 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.478145 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.481070 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:46.481076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:46.481079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:46.481109 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:46.483927 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.484654 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.484658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.484661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.484834 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.486044 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.486048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.486052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.486404 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:46.486558 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:46.486977 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:15:46.487193 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:15:46.487661 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.468652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.66:35217)
0927 22:15:46.471365 (+ 2713us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:46.471394 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:46.471611 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:46.472010 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:46.472018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:46.472850 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:46.473358 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.473375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.474616 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.474625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.477244 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.477266 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.478230 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.478237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.478545 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.479435 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:46.479460 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:46.480867 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:46.484056 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.484061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.484065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.484507 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.484974 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.484978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.484980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.485145 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.486530 (+ 1385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:46.486534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:46.486709 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 22:15:46.486881 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:15:46.487198 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2505,"thread_start_us":2319,"threads_started":1}
I20260927 22:15:46.488032 31841 ts_manager.cc:194] Registered new tserver with Master: 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555)
I20260927 22:15:46.489039 31841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:59453
I20260927 22:15:46.491319 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 32053
I20260927 22:15:46.491503 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:46.491984 24245 external_mini_cluster.cc:1428] Setting key 875d9ef82ceeb53d5ca73322059dfbcf
WARNING: no policy specified for oryx/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
I20260927 22:15:46.537618 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38437
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:46.856222 32192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:46.856498 32192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:46.856596 32192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:46.864497 32192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:46.864640 32192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:46.864799 32192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:15:46.864929 32192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:46.874790 32192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.173.126:38437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:46.876144 32192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:46.877532 32192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:46.890302 32197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:46.890237 32198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:46.890229 32200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:46.891347 32192 server_base.cc:1034] running on GCE node
I20260927 22:15:46.891928 32192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:46.892822 32192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:46.894080 32192 hybrid_clock.cc:648] HybridClock initialized: now 1790547346893988 us; error 42 us; skew 500 ppm
I20260927 22:15:46.894398 32192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:46 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547346, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:46.897449 32192 init.cc:377] Logged in from keytab as oryx/127.23.173.67@KRBTEST.COM (short username oryx)
I20260927 22:15:46.898897 32192 webserver.cc:460] Webserver started at http://127.23.173.67:40263/ using document root <none> and password file <none>
I20260927 22:15:46.899586 32192 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:46.899719 32192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:46.900033 32192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:46.903100 32192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "5d63e5a107ed48f893aa02c861c0dac1"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "f1dfb2d714467ce8d483050dd4e18313"
server_key_iv: "cb1c66e5162927ef4b94823036bf8daf"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.903875 32192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "5d63e5a107ed48f893aa02c861c0dac1"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "f1dfb2d714467ce8d483050dd4e18313"
server_key_iv: "cb1c66e5162927ef4b94823036bf8daf"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.909046 32192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:15:46.912427 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:46.913699 32192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:46.913873 32192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "5d63e5a107ed48f893aa02c861c0dac1"
format_stamp: "Formatted at 2026-09-27 22:15:46 on dist-test-slave-3wc3"
server_key: "f1dfb2d714467ce8d483050dd4e18313"
server_key_iv: "cb1c66e5162927ef4b94823036bf8daf"
server_key_version: "encryptionkey@0"
I20260927 22:15:46.914175 32192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:46.940469 32192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:46.944893 32192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:46.945173 32192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:46.947896 32192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:46.947999 32192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:46.948172 32192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:46.948256 32192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:15:46 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547346, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:46.970700 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.948570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34659 (local address 127.23.173.126:38437)
0927 22:15:46.948802 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:46.948807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:46.950280 (+ 1473us) server_negotiation.cc:407] Connection header received
0927 22:15:46.952267 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:46.952277 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:46.952390 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:46.952505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:46.953992 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.955304 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.958021 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.958819 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.958978 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.963715 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:46.963722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:46.963727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:46.963774 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:46.966651 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.967705 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.967712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.967716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.967913 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.968434 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.968442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.968448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.968955 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:46.969131 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:46.969859 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 22:15:46.970180 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:15:46.971251 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.949149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.67:34659)
0927 22:15:46.950088 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:46.950137 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:46.951992 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:46.952773 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:46.952785 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:46.953262 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:46.953793 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.953815 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.955515 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.955522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.957853 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.957865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.959071 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.959078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.959293 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.960026 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:46.960052 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:46.963462 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:46.966811 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.966825 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.966841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.967541 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.968088 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.968093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.968099 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.968316 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.969090 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:46.969096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:46.969417 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 22:15:46.969639 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 22:15:46.970090 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":393,"threads_started":1}
I20260927 22:15:46.992372 32049 heartbeater.cc:503] Master 127.23.173.126:38437 was elected leader, sending a full tablet report...
I20260927 22:15:46.995055 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.977293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:45573 (local address 127.23.173.126:38437)
0927 22:15:46.977510 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:46.977518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:46.978136 (+ 618us) server_negotiation.cc:407] Connection header received
0927 22:15:46.978401 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:46.978407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:46.978494 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:46.978604 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:46.981687 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.983371 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.985353 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.985791 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.985920 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.988143 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:46.988153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:46.988158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:46.988214 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:46.991346 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.992331 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.992340 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.992346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.992543 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:46.993198 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:46.993206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:46.993211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:46.993683 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:46.993879 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:46.994392 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 22:15:46.994630 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:15:46.995301 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:46.977353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.23.173.67:45573)
0927 22:15:46.978032 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:46.978051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:46.978236 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:46.978624 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:46.978628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:46.979271 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:46.981431 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.981446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.983670 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.983677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.985212 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:46.985225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:46.986013 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:46.986021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:46.986209 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:46.986877 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:46.986910 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:46.987899 (+ 989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:46.991502 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.991507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.991511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.992136 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.992884 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:46.992896 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:46.992901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:46.993069 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:46.993834 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:46.993839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:46.994074 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 22:15:46.994251 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 22:15:46.994642 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":345,"threads_started":1}
I20260927 22:15:47.009716 32192 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:46705
I20260927 22:15:47.009743 32326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:46705 every 8 connection(s)
I20260927 22:15:47.010965 32192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:47.012996 32327 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38437
I20260927 22:15:47.013232 32327 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:47.013860 32327 heartbeater.cc:511] Master 127.23.173.126:38437 requested a full tablet report, sending...
I20260927 22:15:47.015262 31841 ts_manager.cc:194] Registered new tserver with Master: 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:47.016062 31841 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:34659
I20260927 22:15:47.019178 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 32192
I20260927 22:15:47.019376 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:47.019827 24245 external_mini_cluster.cc:1428] Setting key dbf598fd3e6c56c2fea92f27fecba939
I20260927 22:15:47.022569 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:15:47.070685 31937 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:47.060708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.23.173.126:38437)
0927 22:15:47.060948 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.060955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.061078 (+ 123us) server_negotiation.cc:407] Connection header received
0927 22:15:47.061255 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.061259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.061329 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.061417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:47.062757 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.063546 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.064782 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.065170 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.065276 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.070384 (+ 5108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55564: BlockingRecv error: recv got EOF from 127.0.0.1:55564 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:47.086115 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.073616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.23.173.126:38437)
0927 22:15:47.073806 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.073811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.074198 (+ 387us) server_negotiation.cc:407] Connection header received
0927 22:15:47.074327 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.074331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.074416 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.074494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:47.075552 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.076215 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.077352 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.077764 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.077852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.081448 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:47.081453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:47.081456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:47.081485 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:47.083840 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.084541 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.084544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.084565 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.084682 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.085034 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.085037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.085039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.085270 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:47.085486 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.085663 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.085823 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:15:47.089792 31841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:47.096760 31841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:47.121125 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.109032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:46705 (local address 127.0.0.1:37596)
0927 22:15:47.112636 (+ 3604us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.112650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.112814 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.113551 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.113562 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.113594 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.114267 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.114279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.116078 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.116081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.117943 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.118021 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.120308 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.120316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.120528 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.120585 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.120717 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.120862 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3508,"thread_start_us":1111,"threads_started":1}
I20260927 22:15:47.121872 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.109899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.23.173.67:46705)
0927 22:15:47.112172 (+ 2273us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.112184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.112772 (+ 588us) server_negotiation.cc:407] Connection header received
0927 22:15:47.112986 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.112997 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.113275 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.113461 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.114521 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.115932 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.118210 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.119770 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.119876 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.120125 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.121402 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.121577 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2051,"thread_start_us":2274,"threads_started":1}
I20260927 22:15:47.122309 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.108905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:46555 (local address 127.0.0.1:53082)
0927 22:15:47.110541 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.110585 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.110746 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.115370 (+ 4624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.115374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.115405 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.115829 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.115839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.117653 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.117659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.119675 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.119687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.121428 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.121434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.121587 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.121626 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.121695 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.121837 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":57,"thread_start_us":4141,"threads_started":1}
I20260927 22:15:47.123265 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.110968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:34367 (local address 127.0.0.1:43538)
0927 22:15:47.113282 (+ 2314us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.113299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.113407 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.115055 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.115060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.115086 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.115475 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.115488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.117425 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.117431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.119893 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.119906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.122491 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.122500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.122677 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.122707 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.122788 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.122908 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"mutex_wait_us":1,"thread_start_us":1430,"threads_started":1}
I20260927 22:15:47.123354 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.109076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.23.173.66:46555)
0927 22:15:47.113613 (+ 4537us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.113625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.113672 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:15:47.113783 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.113792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.114081 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.115360 (+ 1279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.116018 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.117416 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.119813 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.121201 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.122637 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.122840 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.122900 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.123064 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4360,"thread_start_us":4255,"threads_started":1}
I20260927 22:15:47.123574 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.111305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.23.173.65:34367)
0927 22:15:47.114642 (+ 3337us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.114648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.114674 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:15:47.114741 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.114746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.114889 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.115070 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.115669 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.117120 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.120040 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.121984 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.122127 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.122343 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.123119 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.123307 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3199,"thread_start_us":3050,"threads_started":1}
I20260927 22:15:47.125902 32261 tablet_service.cc:1467] Processing CreateTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 (DEFAULT_TABLE table=test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.126187 32118 tablet_service.cc:1467] Processing CreateTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 (DEFAULT_TABLE table=test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.127522 32118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360bc7e00fd7491d9451f52d860b6fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.127030 31981 tablet_service.cc:1467] Processing CreateTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 (DEFAULT_TABLE table=test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.127924 32261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360bc7e00fd7491d9451f52d860b6fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.128307 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360bc7e00fd7491d9451f52d860b6fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.138093 32351 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Bootstrap starting.
I20260927 22:15:47.141616 32351 tablet_bootstrap.cc:654] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.142274 32352 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Bootstrap starting.
I20260927 22:15:47.143100 32351 log.cc:824] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:47.143953 32353 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Bootstrap starting.
I20260927 22:15:47.146306 32352 tablet_bootstrap.cc:654] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.146795 32353 tablet_bootstrap.cc:654] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.147433 32352 log.cc:824] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:47.147753 32353 log.cc:824] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:47.148655 32351 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: No bootstrap required, opened a new log
I20260927 22:15:47.148922 32351 ts_tablet_manager.cc:1397] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260927 22:15:47.151069 32353 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: No bootstrap required, opened a new log
I20260927 22:15:47.151206 32352 tablet_bootstrap.cc:492] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: No bootstrap required, opened a new log
I20260927 22:15:47.151331 32353 ts_tablet_manager.cc:1397] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:15:47.151458 32352 ts_tablet_manager.cc:1397] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 22:15:47.157922 32353 raft_consensus.cc:348] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.158346 32353 raft_consensus.cc:374] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.158447 32353 raft_consensus.cc:729] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d63e5a107ed48f893aa02c861c0dac1, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.159298 32353 consensus_queue.cc:260] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.159538 32351 raft_consensus.cc:348] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.159904 32351 raft_consensus.cc:374] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.159986 32351 raft_consensus.cc:729] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.160631 32351 consensus_queue.cc:260] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.161130 32327 heartbeater.cc:503] Master 127.23.173.126:38437 was elected leader, sending a full tablet report...
I20260927 22:15:47.161751 32353 ts_tablet_manager.cc:1428] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260927 22:15:47.162305 32352 raft_consensus.cc:348] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.162758 32352 raft_consensus.cc:374] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.162865 32352 raft_consensus.cc:729] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.163456 32352 consensus_queue.cc:260] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.163720 32351 ts_tablet_manager.cc:1428] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260927 22:15:47.164160 32188 heartbeater.cc:503] Master 127.23.173.126:38437 was elected leader, sending a full tablet report...
I20260927 22:15:47.166193 32352 ts_tablet_manager.cc:1428] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260927 22:15:47.186960 32358 raft_consensus.cc:482] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:47.187306 32358 raft_consensus.cc:504] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.189356 32358 leader_election.cc:288] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705), acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367)
I20260927 22:15:47.195808 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.189698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38615 (local address 127.23.173.67:46705)
0927 22:15:47.189900 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.189905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.190351 (+ 446us) server_negotiation.cc:407] Connection header received
0927 22:15:47.190684 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.190690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.190781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.190835 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.191515 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.192258 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.193986 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.194881 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.195046 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.195123 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.195419 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.195537 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:15:47.195969 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.189516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:46705 (local address 127.23.173.66:38615)
0927 22:15:47.190254 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.190274 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.190479 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.190998 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.191001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.191022 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.191373 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.191381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.192395 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.192401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.193806 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.193820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.195084 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.195090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.195263 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.195287 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.195393 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.195537 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":348,"threads_started":1}
I20260927 22:15:47.196939 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.190600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36813 (local address 127.23.173.65:34367)
0927 22:15:47.190805 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.190810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.191237 (+ 427us) server_negotiation.cc:407] Connection header received
0927 22:15:47.191480 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.191494 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.191563 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.191678 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.192269 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.193044 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.194975 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.195927 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.196084 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.196174 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.196575 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.196700 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:15:47.196975 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.190306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:34367 (local address 127.23.173.66:36813)
0927 22:15:47.191115 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.191130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.191319 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.191685 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.191690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.191722 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.192115 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.192124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.193200 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.193214 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.194764 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.194777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.196235 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.196243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.196416 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.196439 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.196546 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.196668 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":291,"threads_started":1}
I20260927 22:15:47.196950 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360bc7e00fd7491d9451f52d860b6fa8" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1" is_pre_election: true
I20260927 22:15:47.197568 32282 raft_consensus.cc:2454] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 0.
I20260927 22:15:47.197989 32004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360bc7e00fd7491d9451f52d860b6fa8" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc83231bddf4f5a867ade4270bb2f5c" is_pre_election: true
I20260927 22:15:47.198297 32071 leader_election.cc:302] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, 8048118b5cee4c2ca00975cc19f7db40; no voters:
I20260927 22:15:47.198515 32004 raft_consensus.cc:2454] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 0.
I20260927 22:15:47.198746 32358 raft_consensus.cc:2781] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:47.198923 32358 raft_consensus.cc:482] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:47.199086 32358 raft_consensus.cc:3037] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.201749 32358 raft_consensus.cc:504] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.202466 32358 leader_election.cc:288] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 election: Requested vote from peers 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705), acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367)
I20260927 22:15:47.202870 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360bc7e00fd7491d9451f52d860b6fa8" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1"
I20260927 22:15:47.203011 32004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360bc7e00fd7491d9451f52d860b6fa8" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc83231bddf4f5a867ade4270bb2f5c"
I20260927 22:15:47.203092 32282 raft_consensus.cc:3037] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.203224 32004 raft_consensus.cc:3037] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.206405 32282 raft_consensus.cc:2454] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 1.
I20260927 22:15:47.206426 32004 raft_consensus.cc:2454] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 1.
I20260927 22:15:47.206800 32071 leader_election.cc:302] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, 8048118b5cee4c2ca00975cc19f7db40; no voters:
I20260927 22:15:47.207103 32358 raft_consensus.cc:2781] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:47.207705 32358 raft_consensus.cc:686] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 LEADER]: Becoming Leader. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Running, Role: LEADER
I20260927 22:15:47.208176 32358 consensus_queue.cc:237] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } }
I20260927 22:15:47.212956 31840 catalog_manager.cc:5495] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66). New cstate: current_term: 1 leader_uuid: "8048118b5cee4c2ca00975cc19f7db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } health_report { overall_health: UNKNOWN } } }
W20260927 22:15:47.234611 32050 tablet.cc:2369] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:15:47.235728 32189 tablet.cc:2369] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:15:47.243932 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.239128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.23.173.66:46555)
0927 22:15:47.239313 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.239318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.239440 (+ 122us) server_negotiation.cc:407] Connection header received
0927 22:15:47.239624 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.239628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.239705 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.239829 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:47.240126 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.240907 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.242141 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.242536 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.242627 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.242984 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.243507 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.243614 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.243679 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.243738 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:15:47.251914 32004 raft_consensus.cc:1261] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Refusing update from remote peer 8048118b5cee4c2ca00975cc19f7db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.251914 32282 raft_consensus.cc:1261] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Refusing update from remote peer 8048118b5cee4c2ca00975cc19f7db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.252833 32362 consensus_queue.cc:1035] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:47.253181 32358 consensus_queue.cc:1035] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:15:47.262557 32328 tablet.cc:2369] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:15:47.278970 31840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:47.279237 31840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:47.282706 31840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Sending DeleteTablet for 3 replicas of tablet 360bc7e00fd7491d9451f52d860b6fa8
I20260927 22:15:47.283766 32118 tablet_service.cc:1514] Processing DeleteTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260927 22:15:47.283864 32261 tablet_service.cc:1514] Processing DeleteTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37596
I20260927 22:15:47.283959 31981 tablet_service.cc:1514] Processing DeleteTablet for tablet 360bc7e00fd7491d9451f52d860b6fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:43538
I20260927 22:15:47.286700 31840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:47.287472 31840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:47.296532 31978 tablet_service.cc:1467] Processing CreateTablet for tablet c344e20bf8094ff49bf0007b90a23f92 (DEFAULT_TABLE table=test-table [id=4c49c5bb7077413fa5e32328e607e7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.297255 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c344e20bf8094ff49bf0007b90a23f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.300709 32117 tablet_service.cc:1467] Processing CreateTablet for tablet c344e20bf8094ff49bf0007b90a23f92 (DEFAULT_TABLE table=test-table [id=4c49c5bb7077413fa5e32328e607e7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.301395 32117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c344e20bf8094ff49bf0007b90a23f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.302575 32376 tablet_replica.cc:330] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: stopping tablet replica
I20260927 22:15:47.303160 32376 raft_consensus.cc:2229] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:47.303069 32260 tablet_service.cc:1467] Processing CreateTablet for tablet c344e20bf8094ff49bf0007b90a23f92 (DEFAULT_TABLE table=test-table [id=4c49c5bb7077413fa5e32328e607e7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.303758 32260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c344e20bf8094ff49bf0007b90a23f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.304023 32376 raft_consensus.cc:2258] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.306354 32376 ts_tablet_manager.cc:1905] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.312644 32353 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Bootstrap starting.
I20260927 22:15:47.313678 32351 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Bootstrap starting.
I20260927 22:15:47.314349 32353 tablet_bootstrap.cc:654] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.315832 32351 tablet_bootstrap.cc:654] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.319546 32376 ts_tablet_manager.cc:1918] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:47.320778 32376 log.cc:1196] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/360bc7e00fd7491d9451f52d860b6fa8
I20260927 22:15:47.321630 32376 ts_tablet_manager.cc:1939] T 360bc7e00fd7491d9451f52d860b6fa8 P 8048118b5cee4c2ca00975cc19f7db40: Deleting consensus metadata
I20260927 22:15:47.322729 32352 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Bootstrap starting.
I20260927 22:15:47.324467 32352 tablet_bootstrap.cc:654] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.324715 31829 catalog_manager.cc:4841] TS 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555): tablet 360bc7e00fd7491d9451f52d860b6fa8 (table test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]) successfully deleted
I20260927 22:15:47.325026 32377 tablet_replica.cc:330] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: stopping tablet replica
I20260927 22:15:47.325495 32377 raft_consensus.cc:2229] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:47.326007 32377 raft_consensus.cc:2258] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.329305 32353 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: No bootstrap required, opened a new log
I20260927 22:15:47.329545 32353 ts_tablet_manager.cc:1397] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.002s
I20260927 22:15:47.330101 32377 ts_tablet_manager.cc:1905] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.331019 32353 raft_consensus.cc:348] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.331393 32353 raft_consensus.cc:374] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.331549 32353 raft_consensus.cc:729] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d63e5a107ed48f893aa02c861c0dac1, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.331930 32353 consensus_queue.cc:260] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.333446 32353 ts_tablet_manager.cc:1428] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:15:47.334472 32375 tablet_replica.cc:330] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: stopping tablet replica
I20260927 22:15:47.334916 32375 raft_consensus.cc:2229] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:47.335441 32375 raft_consensus.cc:2258] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.337987 32375 ts_tablet_manager.cc:1905] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.338150 32351 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: No bootstrap required, opened a new log
I20260927 22:15:47.338332 32351 ts_tablet_manager.cc:1397] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.002s
I20260927 22:15:47.339171 32352 tablet_bootstrap.cc:492] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: No bootstrap required, opened a new log
I20260927 22:15:47.339160 32351 raft_consensus.cc:348] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.339325 32352 ts_tablet_manager.cc:1397] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260927 22:15:47.339396 32351 raft_consensus.cc:374] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.339495 32351 raft_consensus.cc:729] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.339785 32351 consensus_queue.cc:260] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.340119 32352 raft_consensus.cc:348] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.340339 32352 raft_consensus.cc:374] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.340432 32352 raft_consensus.cc:729] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.341236 32351 ts_tablet_manager.cc:1428] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:47.340713 32352 consensus_queue.cc:260] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.345468 32375 ts_tablet_manager.cc:1918] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:47.345665 32375 log.cc:1196] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/360bc7e00fd7491d9451f52d860b6fa8
I20260927 22:15:47.346324 32375 ts_tablet_manager.cc:1939] T 360bc7e00fd7491d9451f52d860b6fa8 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting consensus metadata
I20260927 22:15:47.347059 32377 ts_tablet_manager.cc:1918] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:47.347229 32377 log.cc:1196] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/360bc7e00fd7491d9451f52d860b6fa8
I20260927 22:15:47.347645 32352 ts_tablet_manager.cc:1428] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260927 22:15:47.347738 32377 ts_tablet_manager.cc:1939] T 360bc7e00fd7491d9451f52d860b6fa8 P acc83231bddf4f5a867ade4270bb2f5c: Deleting consensus metadata
I20260927 22:15:47.348433 31828 catalog_manager.cc:4841] TS 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705): tablet 360bc7e00fd7491d9451f52d860b6fa8 (table test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]) successfully deleted
I20260927 22:15:47.349881 31828 catalog_manager.cc:4841] TS acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367): tablet 360bc7e00fd7491d9451f52d860b6fa8 (table test-table [id=22d02a0819ae4e6dbc4de8e7d6b616cc]) successfully deleted
I20260927 22:15:47.463809 32359 raft_consensus.cc:482] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:47.464093 32359 raft_consensus.cc:504] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.465723 32359 leader_election.cc:288] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:47.474138 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.466616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:35783 (local address 127.23.173.67:46705)
0927 22:15:47.466904 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.466910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.467342 (+ 432us) server_negotiation.cc:407] Connection header received
0927 22:15:47.467678 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.467688 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.467789 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.467867 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.468940 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.469998 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.471981 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.472994 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.473109 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.473163 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.473708 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.473831 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 22:15:47.474180 32382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.466032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:46705 (local address 127.23.173.65:35783)
0927 22:15:47.467217 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.467243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.467488 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.468120 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.468127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.468161 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.468777 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.468788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.470161 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.470166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.471774 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.471790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.473384 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.473388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.473534 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.473571 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.473679 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.473823 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":2483,"threads_started":1}
I20260927 22:15:47.475071 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c344e20bf8094ff49bf0007b90a23f92" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1" is_pre_election: true
I20260927 22:15:47.475318 32282 raft_consensus.cc:2454] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:47.475790 31931 leader_election.cc:302] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:47.476221 32359 raft_consensus.cc:2781] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:47.476377 32359 raft_consensus.cc:482] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:47.476476 32359 raft_consensus.cc:3037] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.476521 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.466549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:40767 (local address 127.23.173.66:46555)
0927 22:15:47.466745 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.466752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.468934 (+ 2182us) server_negotiation.cc:407] Connection header received
0927 22:15:47.469197 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.469203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.469283 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.469382 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:15:47.470078 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.471275 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.473910 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.475230 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.475394 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.475466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.476051 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.476212 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:15:47.476552 32381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.466067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:46555 (local address 127.23.173.65:40767)
0927 22:15:47.468840 (+ 2773us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.468858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.469027 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.469422 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.469427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.469454 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:15:47.469834 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.469850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.471488 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.471495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.473738 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.473753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.475608 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.475615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.475812 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.475848 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.475947 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.476078 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2620,"thread_start_us":1140,"threads_started":1}
I20260927 22:15:47.477550 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c344e20bf8094ff49bf0007b90a23f92" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40" is_pre_election: true
I20260927 22:15:47.478060 32138 raft_consensus.cc:2454] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:47.479169 32359 raft_consensus.cc:504] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.479786 32359 leader_election.cc:288] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Requested vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:47.480213 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c344e20bf8094ff49bf0007b90a23f92" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40"
I20260927 22:15:47.480340 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c344e20bf8094ff49bf0007b90a23f92" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1"
I20260927 22:15:47.480437 32138 raft_consensus.cc:3037] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.480543 32282 raft_consensus.cc:3037] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.483428 32282 raft_consensus.cc:2454] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:47.483786 31931 leader_election.cc:302] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:47.484086 32359 raft_consensus.cc:2781] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:47.484202 32138 raft_consensus.cc:2454] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:47.484929 32359 raft_consensus.cc:686] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 1 LEADER]: Becoming Leader. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Running, Role: LEADER
I20260927 22:15:47.485337 32359 consensus_queue.cc:237] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.489215 31842 catalog_manager.cc:5495] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c reported cstate change: term changed from 0 to 1, leader changed from <none> to acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65). New cstate: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547345, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.126@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:47.547715 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.532502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38437 (local address 127.0.0.1:55586)
0927 22:15:47.533171 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.533196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.533324 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.533691 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.533694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.534370 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:47.534646 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.534655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.536275 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.536278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.537257 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.537266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.538296 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.538301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.538410 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.539130 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:47.539147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:47.541476 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:47.545343 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:47.545353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:47.545359 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:47.545791 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:47.546186 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:47.546190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:47.546194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:47.546321 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:47.546913 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:47.546919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:47.547066 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.547198 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.547383 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":373,"threads_started":1}
I20260927 22:15:47.548002 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.532486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.23.173.126:38437)
0927 22:15:47.532771 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.532780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.533302 (+ 522us) server_negotiation.cc:407] Connection header received
0927 22:15:47.533448 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.533454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.533539 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.533670 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:47.534808 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.536036 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.537405 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.538102 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.538241 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.541677 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:47.541684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:47.541689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:47.541727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:47.545212 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.545899 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.545905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.545909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.546079 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.546419 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.546424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.546428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.546782 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:47.546944 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.547333 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.547561 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 22:15:47.553084 31842 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:55586:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:15:47.554265 31842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:47.561218 31978 tablet_service.cc:1467] Processing CreateTablet for tablet 4d554a8da3bc4713850834ec4322f89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f6ededdc6094dd08b6b67432851714e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:15:47.561296 32117 tablet_service.cc:1467] Processing CreateTablet for tablet 4d554a8da3bc4713850834ec4322f89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f6ededdc6094dd08b6b67432851714e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:15:47.561784 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d554a8da3bc4713850834ec4322f89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.561823 32117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d554a8da3bc4713850834ec4322f89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.567953 32351 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: Bootstrap starting.
I20260927 22:15:47.568975 32352 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: Bootstrap starting.
I20260927 22:15:47.569621 32351 tablet_bootstrap.cc:654] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.569971 32261 tablet_service.cc:1467] Processing CreateTablet for tablet 4d554a8da3bc4713850834ec4322f89d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f6ededdc6094dd08b6b67432851714e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:15:47.570537 32261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d554a8da3bc4713850834ec4322f89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.570757 32352 tablet_bootstrap.cc:654] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.578291 32353 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: Bootstrap starting.
I20260927 22:15:47.580495 32353 tablet_bootstrap.cc:654] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.582038 32352 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: No bootstrap required, opened a new log
I20260927 22:15:47.582219 32352 ts_tablet_manager.cc:1397] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 22:15:47.582640 32351 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: No bootstrap required, opened a new log
I20260927 22:15:47.582938 32351 ts_tablet_manager.cc:1397] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260927 22:15:47.582989 32352 raft_consensus.cc:348] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.583253 32352 raft_consensus.cc:374] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.583343 32352 raft_consensus.cc:729] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.583617 32352 consensus_queue.cc:260] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.584065 32359 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.584357 32359 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.584911 32352 ts_tablet_manager.cc:1428] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:47.585400 32351 raft_consensus.cc:348] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.586208 32351 raft_consensus.cc:374] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.586338 32351 raft_consensus.cc:729] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.586701 32351 consensus_queue.cc:260] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.587320 32351 ts_tablet_manager.cc:1428] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:15:47.587204 32370 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.587599 32370 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.590932 32353 tablet_bootstrap.cc:492] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: No bootstrap required, opened a new log
I20260927 22:15:47.591117 32353 ts_tablet_manager.cc:1397] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260927 22:15:47.591833 32353 raft_consensus.cc:348] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.592042 32353 raft_consensus.cc:374] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.592141 32353 raft_consensus.cc:729] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d63e5a107ed48f893aa02c861c0dac1, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.592355 32353 consensus_queue.cc:260] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.592891 32353 ts_tablet_manager.cc:1428] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:15:47.592765 32357 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.593091 32357 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.642247 32359 raft_consensus.cc:482] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:47.642485 32359 raft_consensus.cc:504] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.643247 32359 leader_election.cc:288] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:47.643807 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d554a8da3bc4713850834ec4322f89d" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40" is_pre_election: true
I20260927 22:15:47.643898 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d554a8da3bc4713850834ec4322f89d" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1" is_pre_election: true
I20260927 22:15:47.644078 32138 raft_consensus.cc:2454] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:47.644209 32282 raft_consensus.cc:2454] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:47.644553 31932 leader_election.cc:302] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8048118b5cee4c2ca00975cc19f7db40, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:47.644858 32359 raft_consensus.cc:2781] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:47.644984 32359 raft_consensus.cc:482] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:47.645112 32359 raft_consensus.cc:3037] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.647215 32359 raft_consensus.cc:504] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.647848 32359 leader_election.cc:288] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Requested vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:47.648144 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d554a8da3bc4713850834ec4322f89d" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40"
I20260927 22:15:47.648393 32138 raft_consensus.cc:3037] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.648391 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d554a8da3bc4713850834ec4322f89d" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1"
I20260927 22:15:47.648630 32282 raft_consensus.cc:3037] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:47.651602 32282 raft_consensus.cc:2454] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:47.651607 32138 raft_consensus.cc:2454] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:47.651994 31932 leader_election.cc:302] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8048118b5cee4c2ca00975cc19f7db40, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:47.652259 32359 raft_consensus.cc:2781] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:47.652421 32359 raft_consensus.cc:686] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [term 1 LEADER]: Becoming Leader. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Running, Role: LEADER
I20260927 22:15:47.652657 32359 consensus_queue.cc:237] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.654460 32383 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: TxnStatusTablet state changed. Reason: New leader acc83231bddf4f5a867ade4270bb2f5c. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.654879 32383 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:15:47.655612 32395 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:15:47.655817 32395 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:15:47.656416 31842 catalog_manager.cc:5495] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c reported cstate change: term changed from 0 to 1, leader changed from <none> to acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65). New cstate: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:47.678818 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.670958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:34367 (local address 127.0.0.1:43540)
0927 22:15:47.671161 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.671181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.671323 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.672051 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.672057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.672071 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:47.672257 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.672268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.673915 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.673921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.675576 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.675591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.676663 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.676669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.676860 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.676940 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.678082 (+ 1142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.678108 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.678208 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.678362 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:15:47.678838 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.670967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.23.173.65:34367)
0927 22:15:47.671159 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.671165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.671508 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:15:47.671601 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.671612 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.671728 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.671862 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:47.672422 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.673735 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.675756 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.676377 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.676530 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.677168 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.677861 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.678049 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.678361 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.678449 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:15:47.683995 32282 raft_consensus.cc:1261] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.684381 32138 raft_consensus.cc:1261] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.684777 32359 consensus_queue.cc:1035] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:47.685177 32383 consensus_queue.cc:1035] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:47.697993 32370 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: TxnStatusTablet state changed. Reason: New leader acc83231bddf4f5a867ade4270bb2f5c. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.698314 32370 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.698158 32357 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: TxnStatusTablet state changed. Reason: New leader acc83231bddf4f5a867ade4270bb2f5c. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.698406 32357 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.701220 32359 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.701521 32359 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:15:47.707779 32383 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.708068 32383 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P acc83231bddf4f5a867ade4270bb2f5c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:15:47.709002 32357 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.709229 32357 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 5d63e5a107ed48f893aa02c861c0dac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.713186 32365 mvcc.cc:205] Tried to move back new op lower bound from 7334081936105869312 to 7334081935987650560. Current Snapshot: MvccSnapshot[applied={T|T < 7334081936105869312}]
I20260927 22:15:47.714155 32370 tablet_replica.cc:439] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } } }
I20260927 22:15:47.714412 32370 tablet_replica.cc:442] T 4d554a8da3bc4713850834ec4322f89d P 8048118b5cee4c2ca00975cc19f7db40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:15:47.720597 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.711262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.23.173.65:34367)
0927 22:15:47.711469 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.711475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.711493 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:47.711573 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.711578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.711655 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.711750 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:47.713629 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.715884 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.718574 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.719150 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.719370 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.719768 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.719865 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.719972 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.720104 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.720185 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:15:47.737226 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.730989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:41119 (local address 127.23.173.65:34367)
0927 22:15:47.731221 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.731229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.731727 (+ 498us) server_negotiation.cc:407] Connection header received
0927 22:15:47.731923 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.731928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.731990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.732086 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:47.732594 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.733328 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.735300 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.735713 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.735879 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.736334 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.736493 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.736576 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.736821 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.736907 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:15:47.737874 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.730775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:34367 (local address 127.23.173.65:41119)
0927 22:15:47.731636 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:47.731651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:47.731807 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:47.732096 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:47.732100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:47.732109 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:47.732455 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.732463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.733462 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.733467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.735153 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:47.735168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.735900 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.735907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:47.736105 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.736165 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:47.736665 (+ 500us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:47.736673 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:15:47.737458 (+ 785us) client_negotiation.cc:240] Negotiation successful
0927 22:15:47.737600 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":579,"threads_started":1}
I20260927 22:15:47.748317 32282 raft_consensus.cc:1261] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.748895 32137 raft_consensus.cc:1261] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:47.748847 32383 consensus_queue.cc:1035] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:47.749723 32359 consensus_queue.cc:1035] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:47.764917 32367 mvcc.cc:205] Tried to move back new op lower bound from 7334081936371830784 to 7334081935304630272. Current Snapshot: MvccSnapshot[applied={T|T < 7334081936371830784}]
I20260927 22:15:47.836380 31840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:47.836632 31840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:47.839924 31840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Sending DeleteTablet for 3 replicas of tablet c344e20bf8094ff49bf0007b90a23f92
I20260927 22:15:47.841006 32117 tablet_service.cc:1514] Processing DeleteTablet for tablet c344e20bf8094ff49bf0007b90a23f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260927 22:15:47.841043 32261 tablet_service.cc:1514] Processing DeleteTablet for tablet c344e20bf8094ff49bf0007b90a23f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37596
I20260927 22:15:47.841557 32375 tablet_replica.cc:330] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: stopping tablet replica
I20260927 22:15:47.841715 31978 tablet_service.cc:1514] Processing DeleteTablet for tablet c344e20bf8094ff49bf0007b90a23f92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:47 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:43538
I20260927 22:15:47.841776 32375 raft_consensus.cc:2229] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:47.842000 32375 raft_consensus.cc:2258] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.842068 32377 tablet_replica.cc:330] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: stopping tablet replica
I20260927 22:15:47.842279 32377 raft_consensus.cc:2229] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:47.842767 32377 raft_consensus.cc:2258] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.843953 32377 ts_tablet_manager.cc:1905] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.847376 32375 ts_tablet_manager.cc:1905] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.851555 32377 ts_tablet_manager.cc:1918] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:47.851712 32377 log.cc:1196] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/c344e20bf8094ff49bf0007b90a23f92
I20260927 22:15:47.852135 32377 ts_tablet_manager.cc:1939] T c344e20bf8094ff49bf0007b90a23f92 P acc83231bddf4f5a867ade4270bb2f5c: Deleting consensus metadata
I20260927 22:15:47.853288 31828 catalog_manager.cc:4841] TS acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367): tablet c344e20bf8094ff49bf0007b90a23f92 (table test-table [id=4c49c5bb7077413fa5e32328e607e7be]) successfully deleted
I20260927 22:15:47.858246 32375 ts_tablet_manager.cc:1918] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:47.858403 32375 log.cc:1196] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/c344e20bf8094ff49bf0007b90a23f92
I20260927 22:15:47.858830 32375 ts_tablet_manager.cc:1939] T c344e20bf8094ff49bf0007b90a23f92 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting consensus metadata
I20260927 22:15:47.860025 31828 catalog_manager.cc:4841] TS 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705): tablet c344e20bf8094ff49bf0007b90a23f92 (table test-table [id=4c49c5bb7077413fa5e32328e607e7be]) successfully deleted
I20260927 22:15:47.860081 32407 tablet_replica.cc:330] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: stopping tablet replica
I20260927 22:15:47.860296 32407 raft_consensus.cc:2229] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:47.860526 32407 raft_consensus.cc:2258] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:47.861613 32407 ts_tablet_manager.cc:1905] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:47.868896 32407 ts_tablet_manager.cc:1918] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:47.869066 32407 log.cc:1196] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/c344e20bf8094ff49bf0007b90a23f92
I20260927 22:15:47.869479 32407 ts_tablet_manager.cc:1939] T c344e20bf8094ff49bf0007b90a23f92 P 8048118b5cee4c2ca00975cc19f7db40: Deleting consensus metadata
I20260927 22:15:47.870805 31829 catalog_manager.cc:4841] TS 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555): tablet c344e20bf8094ff49bf0007b90a23f92 (table test-table [id=4c49c5bb7077413fa5e32328e607e7be]) successfully deleted
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:15:47.916085 31937 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:47.897483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.23.173.126:38437)
0927 22:15:47.897701 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.897708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.898118 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:15:47.898291 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.898297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.898391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.898516 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:47.899898 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.901054 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.902527 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.903232 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.903359 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.915801 (+ 12442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55592: BlockingRecv error: recv got EOF from 127.0.0.1:55592 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:15:47 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM
I20260927 22:15:47.934144 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:47.919120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.23.173.126:38437)
0927 22:15:47.919302 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:47.919310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:47.919631 (+ 321us) server_negotiation.cc:407] Connection header received
0927 22:15:47.919806 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:47.919811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:47.919896 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:47.920003 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:47.921107 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.922236 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.924111 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:47.924532 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:47.924627 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:47.928867 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:47.928872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:47.928875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:47.928908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:47.931550 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.932186 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.932193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.932201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.932321 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:47.932689 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:47.932702 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:47.932706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:47.932976 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:47.933222 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:47.933528 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:15:47.933759 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:15:47.937528 31840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:47.938186 31840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:47.945057 31978 tablet_service.cc:1467] Processing CreateTablet for tablet 3280bfc04c9b469d89cb132f97112d27 (DEFAULT_TABLE table=test-table [id=d0abea17cdb3401c9b782f0b97b39073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.945356 32117 tablet_service.cc:1467] Processing CreateTablet for tablet 3280bfc04c9b469d89cb132f97112d27 (DEFAULT_TABLE table=test-table [id=d0abea17cdb3401c9b782f0b97b39073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.945547 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3280bfc04c9b469d89cb132f97112d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.945825 32117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3280bfc04c9b469d89cb132f97112d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.946744 32261 tablet_service.cc:1467] Processing CreateTablet for tablet 3280bfc04c9b469d89cb132f97112d27 (DEFAULT_TABLE table=test-table [id=d0abea17cdb3401c9b782f0b97b39073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:47.947144 32261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3280bfc04c9b469d89cb132f97112d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:47.952612 32352 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Bootstrap starting.
I20260927 22:15:47.953374 32353 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Bootstrap starting.
I20260927 22:15:47.954421 32352 tablet_bootstrap.cc:654] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.954985 32353 tablet_bootstrap.cc:654] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.956218 32351 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Bootstrap starting.
I20260927 22:15:47.957820 32351 tablet_bootstrap.cc:654] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:47.959416 32352 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: No bootstrap required, opened a new log
I20260927 22:15:47.959590 32352 ts_tablet_manager.cc:1397] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 22:15:47.960389 32352 raft_consensus.cc:348] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.960750 32352 raft_consensus.cc:374] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.960866 32352 raft_consensus.cc:729] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.961216 32352 consensus_queue.cc:260] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.961814 32352 ts_tablet_manager.cc:1428] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:47.963969 32353 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: No bootstrap required, opened a new log
I20260927 22:15:47.964159 32351 tablet_bootstrap.cc:492] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: No bootstrap required, opened a new log
I20260927 22:15:47.964110 32353 ts_tablet_manager.cc:1397] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 22:15:47.964284 32351 ts_tablet_manager.cc:1397] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260927 22:15:47.964957 32353 raft_consensus.cc:348] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.964977 32351 raft_consensus.cc:348] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.965221 32351 raft_consensus.cc:374] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.965210 32353 raft_consensus.cc:374] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:47.965304 32351 raft_consensus.cc:729] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.965309 32353 raft_consensus.cc:729] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d63e5a107ed48f893aa02c861c0dac1, State: Initialized, Role: FOLLOWER
I20260927 22:15:47.965505 32351 consensus_queue.cc:260] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.965619 32353 consensus_queue.cc:260] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:47.966028 32351 ts_tablet_manager.cc:1428] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:15:47.966831 32353 ts_tablet_manager.cc:1428] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 22:15:48.031639 32359 raft_consensus.cc:482] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:48.031860 32359 raft_consensus.cc:504] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.032596 32359 leader_election.cc:288] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:48.033098 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3280bfc04c9b469d89cb132f97112d27" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40" is_pre_election: true
I20260927 22:15:48.033360 32138 raft_consensus.cc:2454] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:48.033296 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3280bfc04c9b469d89cb132f97112d27" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1" is_pre_election: true
I20260927 22:15:48.033551 32281 raft_consensus.cc:2454] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 0.
I20260927 22:15:48.034062 31931 leader_election.cc:302] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:48.034386 32359 raft_consensus.cc:2781] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:48.034534 32359 raft_consensus.cc:482] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:48.034662 32359 raft_consensus.cc:3037] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.037264 32359 raft_consensus.cc:504] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.037901 32359 leader_election.cc:288] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Requested vote from peers 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:48.038324 32138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3280bfc04c9b469d89cb132f97112d27" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8048118b5cee4c2ca00975cc19f7db40"
I20260927 22:15:48.038558 32138 raft_consensus.cc:3037] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.038712 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3280bfc04c9b469d89cb132f97112d27" candidate_uuid: "acc83231bddf4f5a867ade4270bb2f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1"
I20260927 22:15:48.038930 32281 raft_consensus.cc:3037] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.041522 32138 raft_consensus.cc:2454] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:48.041972 31932 leader_election.cc:302] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8048118b5cee4c2ca00975cc19f7db40, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:48.042104 32281 raft_consensus.cc:2454] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc83231bddf4f5a867ade4270bb2f5c in term 1.
I20260927 22:15:48.042418 32359 raft_consensus.cc:2781] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:48.042910 32359 raft_consensus.cc:686] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 1 LEADER]: Becoming Leader. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Running, Role: LEADER
I20260927 22:15:48.043213 32359 consensus_queue.cc:237] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.046072 31842 catalog_manager.cc:5495] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c reported cstate change: term changed from 0 to 1, leader changed from <none> to acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65). New cstate: current_term: 1 leader_uuid: "acc83231bddf4f5a867ade4270bb2f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:48.067013 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.061772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.23.173.65:34367)
0927 22:15:48.062200 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.062206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.062226 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:15:48.062291 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.062297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.062373 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.062470 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:48.062852 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.063745 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.065159 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.065612 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.065789 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.066293 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.066418 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.066525 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.066645 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.066732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260927 22:15:48.069766 32138 raft_consensus.cc:1261] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:48.069846 32281 raft_consensus.cc:1261] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Refusing update from remote peer acc83231bddf4f5a867ade4270bb2f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:48.070562 32359 consensus_queue.cc:1035] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:48.070941 32405 consensus_queue.cc:1035] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:48.089895 31842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:48.090194 31842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:48.093147 31842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Sending DeleteTablet for 3 replicas of tablet 3280bfc04c9b469d89cb132f97112d27
I20260927 22:15:48.094002 32117 tablet_service.cc:1514] Processing DeleteTablet for tablet 3280bfc04c9b469d89cb132f97112d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260927 22:15:48.094007 31978 tablet_service.cc:1514] Processing DeleteTablet for tablet 3280bfc04c9b469d89cb132f97112d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:43538
I20260927 22:15:48.094295 32261 tablet_service.cc:1514] Processing DeleteTablet for tablet 3280bfc04c9b469d89cb132f97112d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37596
I20260927 22:15:48.094633 32375 tablet_replica.cc:330] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: stopping tablet replica
I20260927 22:15:48.094628 32407 tablet_replica.cc:330] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: stopping tablet replica
I20260927 22:15:48.094834 32375 raft_consensus.cc:2229] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:48.094852 32407 raft_consensus.cc:2229] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:48.095070 32375 raft_consensus.cc:2258] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.095119 32407 raft_consensus.cc:2258] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.095429 32377 tablet_replica.cc:330] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: stopping tablet replica
I20260927 22:15:48.095630 32377 raft_consensus.cc:2229] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:48.096000 32377 raft_consensus.cc:2258] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.097046 32377 ts_tablet_manager.cc:1905] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.097318 32375 ts_tablet_manager.cc:1905] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.098567 32407 ts_tablet_manager.cc:1905] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.104000 32377 ts_tablet_manager.cc:1918] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:48.104139 32377 log.cc:1196] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/3280bfc04c9b469d89cb132f97112d27
I20260927 22:15:48.104609 32377 ts_tablet_manager.cc:1939] T 3280bfc04c9b469d89cb132f97112d27 P acc83231bddf4f5a867ade4270bb2f5c: Deleting consensus metadata
I20260927 22:15:48.105509 32375 ts_tablet_manager.cc:1918] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:48.105654 32375 log.cc:1196] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/3280bfc04c9b469d89cb132f97112d27
I20260927 22:15:48.105697 31828 catalog_manager.cc:4841] TS acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367): tablet 3280bfc04c9b469d89cb132f97112d27 (table test-table [id=d0abea17cdb3401c9b782f0b97b39073]) successfully deleted
I20260927 22:15:48.106108 32407 ts_tablet_manager.cc:1918] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:15:48.106127 32375 ts_tablet_manager.cc:1939] T 3280bfc04c9b469d89cb132f97112d27 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting consensus metadata
I20260927 22:15:48.106277 32407 log.cc:1196] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/3280bfc04c9b469d89cb132f97112d27
I20260927 22:15:48.106616 32407 ts_tablet_manager.cc:1939] T 3280bfc04c9b469d89cb132f97112d27 P 8048118b5cee4c2ca00975cc19f7db40: Deleting consensus metadata
I20260927 22:15:48.107257 31828 catalog_manager.cc:4841] TS 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705): tablet 3280bfc04c9b469d89cb132f97112d27 (table test-table [id=d0abea17cdb3401c9b782f0b97b39073]) successfully deleted
I20260927 22:15:48.107692 31829 catalog_manager.cc:4841] TS 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555): tablet 3280bfc04c9b469d89cb132f97112d27 (table test-table [id=d0abea17cdb3401c9b782f0b97b39073]) successfully deleted
I20260927 22:15:48.107371 31842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:15:48.108439 31842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:15:48.115159 31978 tablet_service.cc:1467] Processing CreateTablet for tablet b77dfde8039d4567b20fcf9b66752349 (DEFAULT_TABLE table=test-table [id=042a74777bee4d9296eaea4630082c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:48.115487 32117 tablet_service.cc:1467] Processing CreateTablet for tablet b77dfde8039d4567b20fcf9b66752349 (DEFAULT_TABLE table=test-table [id=042a74777bee4d9296eaea4630082c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:48.115667 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77dfde8039d4567b20fcf9b66752349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:48.115873 32117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77dfde8039d4567b20fcf9b66752349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:48.118780 32261 tablet_service.cc:1467] Processing CreateTablet for tablet b77dfde8039d4567b20fcf9b66752349 (DEFAULT_TABLE table=test-table [id=042a74777bee4d9296eaea4630082c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:15:48.119213 32261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77dfde8039d4567b20fcf9b66752349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:48.122166 32351 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Bootstrap starting.
I20260927 22:15:48.123865 32351 tablet_bootstrap.cc:654] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:48.126157 32353 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Bootstrap starting.
I20260927 22:15:48.126823 32352 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Bootstrap starting.
I20260927 22:15:48.128368 32353 tablet_bootstrap.cc:654] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:48.128538 32352 tablet_bootstrap.cc:654] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:48.135819 32351 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: No bootstrap required, opened a new log
I20260927 22:15:48.135965 32353 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: No bootstrap required, opened a new log
I20260927 22:15:48.135993 32351 ts_tablet_manager.cc:1397] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260927 22:15:48.136129 32353 ts_tablet_manager.cc:1397] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 22:15:48.136839 32353 raft_consensus.cc:348] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.136807 32351 raft_consensus.cc:348] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.137072 32351 raft_consensus.cc:374] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:48.137072 32353 raft_consensus.cc:374] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:48.137145 32351 raft_consensus.cc:729] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Initialized, Role: FOLLOWER
I20260927 22:15:48.137145 32353 raft_consensus.cc:729] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d63e5a107ed48f893aa02c861c0dac1, State: Initialized, Role: FOLLOWER
I20260927 22:15:48.137430 32351 consensus_queue.cc:260] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.137429 32353 consensus_queue.cc:260] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.138060 32353 ts_tablet_manager.cc:1428] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 22:15:48.140256 32352 tablet_bootstrap.cc:492] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: No bootstrap required, opened a new log
I20260927 22:15:48.140501 32352 ts_tablet_manager.cc:1397] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260927 22:15:48.141296 32352 raft_consensus.cc:348] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.141563 32352 raft_consensus.cc:374] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:48.141657 32352 raft_consensus.cc:729] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc83231bddf4f5a867ade4270bb2f5c, State: Initialized, Role: FOLLOWER
I20260927 22:15:48.142279 32351 ts_tablet_manager.cc:1428] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260927 22:15:48.141934 32352 consensus_queue.cc:260] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.143838 32352 ts_tablet_manager.cc:1428] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:15:48.579653 32370 raft_consensus.cc:482] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:15:48.579877 32370 raft_consensus.cc:504] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.580401 32370 leader_election.cc:288] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:48.580960 32004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77dfde8039d4567b20fcf9b66752349" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc83231bddf4f5a867ade4270bb2f5c" is_pre_election: true
I20260927 22:15:48.581045 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77dfde8039d4567b20fcf9b66752349" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1" is_pre_election: true
I20260927 22:15:48.581228 32004 raft_consensus.cc:2454] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 0.
I20260927 22:15:48.581331 32281 raft_consensus.cc:2454] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 0.
I20260927 22:15:48.581779 32071 leader_election.cc:302] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8048118b5cee4c2ca00975cc19f7db40, acc83231bddf4f5a867ade4270bb2f5c; no voters:
I20260927 22:15:48.582086 32370 raft_consensus.cc:2781] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:15:48.582231 32370 raft_consensus.cc:482] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:15:48.582329 32370 raft_consensus.cc:3037] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.584723 32370 raft_consensus.cc:504] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.585239 32370 leader_election.cc:288] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 election: Requested vote from peers acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367), 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705)
I20260927 22:15:48.585645 32004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77dfde8039d4567b20fcf9b66752349" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc83231bddf4f5a867ade4270bb2f5c"
I20260927 22:15:48.585717 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77dfde8039d4567b20fcf9b66752349" candidate_uuid: "8048118b5cee4c2ca00975cc19f7db40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d63e5a107ed48f893aa02c861c0dac1"
I20260927 22:15:48.585871 32004 raft_consensus.cc:3037] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.585893 32281 raft_consensus.cc:3037] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:48.588737 32004 raft_consensus.cc:2454] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 1.
I20260927 22:15:48.588737 32281 raft_consensus.cc:2454] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8048118b5cee4c2ca00975cc19f7db40 in term 1.
I20260927 22:15:48.589408 32071 leader_election.cc:302] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d63e5a107ed48f893aa02c861c0dac1, 8048118b5cee4c2ca00975cc19f7db40; no voters:
I20260927 22:15:48.589658 32370 raft_consensus.cc:2781] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:48.590065 32370 raft_consensus.cc:686] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 LEADER]: Becoming Leader. State: Replica: 8048118b5cee4c2ca00975cc19f7db40, State: Running, Role: LEADER
I20260927 22:15:48.590348 32370 consensus_queue.cc:237] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } }
I20260927 22:15:48.592656 31842 catalog_manager.cc:5495] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66). New cstate: current_term: 1 leader_uuid: "8048118b5cee4c2ca00975cc19f7db40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8048118b5cee4c2ca00975cc19f7db40" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 } health_report { overall_health: UNKNOWN } } }
I20260927 22:15:48.708369 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.703122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.23.173.66:46555)
0927 22:15:48.703673 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.703679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.703808 (+ 129us) server_negotiation.cc:407] Connection header received
0927 22:15:48.703935 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.703939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.704000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.704081 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:48.704506 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.705233 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.706617 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.707004 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.707142 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.707568 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.707689 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.707813 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.707965 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.708049 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260927 22:15:48.722247 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.717174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:37857 (local address 127.23.173.65:34367)
0927 22:15:48.717736 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.717740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.717764 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:15:48.717948 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.717990 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.718070 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.718232 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:48.718591 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.719393 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.720653 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.721069 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.721155 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.721531 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.721629 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.721695 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.721814 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.721870 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":332,"threads_started":1}
I20260927 22:15:48.722280 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.717068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:34367 (local address 127.23.173.66:37857)
0927 22:15:48.717638 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:48.717654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:48.717759 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:48.718243 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:48.718248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:48.718261 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:48.718457 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:48.718464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.719513 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.719517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:48.720539 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:48.720550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.721214 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.721218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:48.721329 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.721379 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.721714 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.721722 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:15:48.721780 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 22:15:48.721874 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":327,"threads_started":1}
I20260927 22:15:48.729604 32281 raft_consensus.cc:1261] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Refusing update from remote peer 8048118b5cee4c2ca00975cc19f7db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:48.729638 32004 raft_consensus.cc:1261] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Refusing update from remote peer 8048118b5cee4c2ca00975cc19f7db40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:15:48.730160 32430 consensus_queue.cc:1035] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d63e5a107ed48f893aa02c861c0dac1" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:48.730485 32370 consensus_queue.cc:1035] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc83231bddf4f5a867ade4270bb2f5c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:15:48.756084 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.750750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:38271 (local address 127.23.173.66:46555)
0927 22:15:48.751150 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.751163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.751478 (+ 315us) server_negotiation.cc:407] Connection header received
0927 22:15:48.751593 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.751596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.751654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.751731 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:15:48.752062 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.752877 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.754692 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.755084 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.755181 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.755532 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.755649 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.755723 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.755844 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.755901 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260927 22:15:48.756234 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.750784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:46555 (local address 127.23.173.65:38271)
0927 22:15:48.751388 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:48.751407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:48.751512 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:48.751735 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:48.751738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:48.751746 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:15:48.751951 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:48.751958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.752997 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.753002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:48.754566 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:48.754578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.755237 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.755241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:48.755378 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.755411 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:15:48.755744 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:15:48.755753 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:15:48.755810 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 22:15:48.755902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":377,"threads_started":1}
I20260927 22:15:48.784525 31840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:48.784762 31840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:15:48.787482 31840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9401860d4bc48d49f515f34658357e0: Sending DeleteTablet for 3 replicas of tablet b77dfde8039d4567b20fcf9b66752349
I20260927 22:15:48.788204 31978 tablet_service.cc:1514] Processing DeleteTablet for tablet b77dfde8039d4567b20fcf9b66752349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:43538
I20260927 22:15:48.788300 32117 tablet_service.cc:1514] Processing DeleteTablet for tablet b77dfde8039d4567b20fcf9b66752349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:53082
I20260927 22:15:48.788341 32261 tablet_service.cc:1514] Processing DeleteTablet for tablet b77dfde8039d4567b20fcf9b66752349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:15:48 UTC) from {username='oryx', principal='oryx/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37596
I20260927 22:15:48.790414 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 31913
I20260927 22:15:48.792490 32456 tablet_replica.cc:330] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: stopping tablet replica
I20260927 22:15:48.792712 32456 raft_consensus.cc:2229] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:48.792762 32454 tablet_replica.cc:330] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: stopping tablet replica
I20260927 22:15:48.792995 32456 raft_consensus.cc:2258] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.793022 32454 raft_consensus.cc:2229] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:15:48.793363 32454 raft_consensus.cc:2258] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.793630 32455 tablet_replica.cc:330] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: stopping tablet replica
I20260927 22:15:48.793819 32455 raft_consensus.cc:2229] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:15:48.794247 32455 raft_consensus.cc:2258] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:15:48.794396 32456 ts_tablet_manager.cc:1905] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.794560 32454 ts_tablet_manager.cc:1905] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.795193 32455 ts_tablet_manager.cc:1905] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:15:48.801371 32456 ts_tablet_manager.cc:1918] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:48.801589 32456 log.cc:1196] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/b77dfde8039d4567b20fcf9b66752349
I20260927 22:15:48.801613 32454 ts_tablet_manager.cc:1918] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:48.801661 32455 ts_tablet_manager.cc:1918] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:15:48.801750 32454 log.cc:1196] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/b77dfde8039d4567b20fcf9b66752349
I20260927 22:15:48.801817 32455 log.cc:1196] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/b77dfde8039d4567b20fcf9b66752349
I20260927 22:15:48.802279 32454 ts_tablet_manager.cc:1939] T b77dfde8039d4567b20fcf9b66752349 P acc83231bddf4f5a867ade4270bb2f5c: Deleting consensus metadata
I20260927 22:15:48.802418 32455 ts_tablet_manager.cc:1939] T b77dfde8039d4567b20fcf9b66752349 P 8048118b5cee4c2ca00975cc19f7db40: Deleting consensus metadata
I20260927 22:15:48.802410 32456 ts_tablet_manager.cc:1939] T b77dfde8039d4567b20fcf9b66752349 P 5d63e5a107ed48f893aa02c861c0dac1: Deleting consensus metadata
I20260927 22:15:48.803479 31828 catalog_manager.cc:4841] TS acc83231bddf4f5a867ade4270bb2f5c (127.23.173.65:34367): tablet b77dfde8039d4567b20fcf9b66752349 (table test-table [id=042a74777bee4d9296eaea4630082c45]) successfully deleted
I20260927 22:15:48.803709 31829 catalog_manager.cc:4841] TS 8048118b5cee4c2ca00975cc19f7db40 (127.23.173.66:46555): tablet b77dfde8039d4567b20fcf9b66752349 (table test-table [id=042a74777bee4d9296eaea4630082c45]) successfully deleted
I20260927 22:15:48.803771 31828 catalog_manager.cc:4841] TS 5d63e5a107ed48f893aa02c861c0dac1 (127.23.173.67:46705): tablet b77dfde8039d4567b20fcf9b66752349 (table test-table [id=042a74777bee4d9296eaea4630082c45]) successfully deleted
Sep 27 22:15:48 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.65@KRBTEST.COM
I20260927 22:15:48.810667 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.791754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.23.173.65:34367)
0927 22:15:48.791929 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.791938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.792975 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 22:15:48.793514 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.793525 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.793617 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.793711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:48.795535 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.796771 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.798658 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.799371 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.799487 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.804992 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:48.805007 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:48.805024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:48.805055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:48.807943 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:48.808559 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:48.808567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:48.808569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:48.808707 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:48.809181 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:48.809186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:48.809191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:48.809534 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:48.809827 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.810179 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.810415 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 22:15:48.811324 32044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:48.957213 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 31913
I20260927 22:15:48.982712 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 32053
Sep 27 22:15:48 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.66@KRBTEST.COM
I20260927 22:15:48.998390 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:48.983510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.23.173.66:46555)
0927 22:15:48.983677 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:48.983681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:48.983698 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:15:48.983842 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:48.983846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:48.983905 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:48.983987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:48.985193 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.985930 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.987186 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:48.987737 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:48.987890 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:48.992812 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:48.992826 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:48.992842 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:48.992874 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:48.996067 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:48.996636 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:48.996643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:48.996645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:48.996773 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:48.997067 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:48.997070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:48.997071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:48.997401 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:48.997616 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:48.997815 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:15:48.998024 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:15:48.999042 32183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:49.123517 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 32053
I20260927 22:15:49.146165 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 32192
Sep 27 22:15:49 dist-test-slave-3wc3 krb5kdc[31797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.173.67@KRBTEST.COM
I20260927 22:15:49.161475 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:49.146914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.23.173.67:46705)
0927 22:15:49.147427 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:49.147431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:49.147446 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:15:49.147501 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:49.147505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:49.147569 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:49.147702 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:49.148914 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:49.149715 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:49.151175 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:49.151608 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:49.151746 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:49.155978 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:49.155993 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:49.156010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:49.156045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:49.158763 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:49.159430 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:49.159438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:49.159440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:49.159564 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:49.159926 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:49.159933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:49.159936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:49.160288 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:49.160604 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:49.160901 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:15:49.161134 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":236,"threads_started":1}
I20260927 22:15:49.162233 32322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:49.258302 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 32192
I20260927 22:15:49.275687 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 31810
I20260927 22:15:49.276885 31882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:49.393910 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 31810
2026-09-27T22:15:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5797 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:49 dist-test-slave-3wc3 krb5kdc[32465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:49 dist-test-slave-3wc3 krb5kdc[32465](info): set up 2 sockets
Sep 27 22:15:49 dist-test-slave-3wc3 krb5kdc[32465](info): commencing operation
krb5kdc: starting...
W20260927 22:15:50.495009 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:50 dist-test-slave-3wc3 krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:50.706779 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:45411
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:51.033775 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:51.034106 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:51.034307 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:51.041519 32478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:51.041621 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:51.041752 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:51.041852 32478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:51.041936 32478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:51.051836 32478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:45411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:51.053108 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:51.054464 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:51.066694 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:51.066694 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:51.067014 32483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:51.067466 32478 server_base.cc:1034] running on GCE node
I20260927 22:15:51.068279 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:51.069658 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:51.070886 32478 hybrid_clock.cc:648] HybridClock initialized: now 1790547351070798 us; error 72 us; skew 500 ppm
I20260927 22:15:51.071287 32478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:15:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1739 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:15:51 dist-test-slave-3wc3 krb5kdc[32492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:15:51 dist-test-slave-3wc3 krb5kdc[32492](info): set up 2 sockets
Sep 27 22:15:51 dist-test-slave-3wc3 krb5kdc[32492](info): commencing operation
krb5kdc: starting...
W20260927 22:15:52.226907 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:15:52 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:15:52.447839 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37215
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:52.766214 32505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:52.766537 32505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:52.766705 32505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:52.773936 32505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:52.774121 32505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:52.774225 32505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:52.774329 32505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:52.774405 32505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:52.784246 32505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:52.785529 32505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:52.786875 32505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:52.798355 32510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:52.798452 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:52.798452 32511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:52.799256 32505 server_base.cc:1034] running on GCE node
I20260927 22:15:52.799960 32505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:52.801280 32505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:52.802502 32505 hybrid_clock.cc:648] HybridClock initialized: now 1790547352802462 us; error 49 us; skew 500 ppm
I20260927 22:15:52.802834 32505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:52 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:52.805923 32505 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:52.807294 32505 webserver.cc:460] Webserver started at http://127.23.173.126:44339/ using document root <none> and password file <none>
I20260927 22:15:52.807891 32505 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:52.808037 32505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:52.808369 32505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:52.811437 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "9af307a6f0e44c29bdc90fad59c62a6b"
format_stamp: "Formatted at 2026-09-27 22:15:52 on dist-test-slave-3wc3"
server_key: "2fb40a12556d4ba20aefaef5ce9f0d57"
server_key_iv: "afe82446c3792fa945c3b77ae6cc6d39"
server_key_version: "encryptionkey@0"
I20260927 22:15:52.812162 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "9af307a6f0e44c29bdc90fad59c62a6b"
format_stamp: "Formatted at 2026-09-27 22:15:52 on dist-test-slave-3wc3"
server_key: "2fb40a12556d4ba20aefaef5ce9f0d57"
server_key_iv: "afe82446c3792fa945c3b77ae6cc6d39"
server_key_version: "encryptionkey@0"
I20260927 22:15:52.817106 32505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:15:52.820508 32520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:52.821796 32505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:52.821964 32505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9af307a6f0e44c29bdc90fad59c62a6b"
format_stamp: "Formatted at 2026-09-27 22:15:52 on dist-test-slave-3wc3"
server_key: "2fb40a12556d4ba20aefaef5ce9f0d57"
server_key_iv: "afe82446c3792fa945c3b77ae6cc6d39"
server_key_version: "encryptionkey@0"
I20260927 22:15:52.822192 32505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:52.873916 32505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:52.878540 32505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:52.878837 32505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:52.897468 32505 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37215
I20260927 22:15:52.897480 32571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37215 every 8 connection(s)
I20260927 22:15:52.898919 32505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:52.900014 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 32505
I20260927 22:15:52.900236 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:15:52.901079 24245 external_mini_cluster.cc:1428] Setting key 059e20387f47618820c584dfe4b5277d
I20260927 22:15:52.903846 32572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:15:52.915596 32572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Bootstrap starting.
I20260927 22:15:52.918465 32572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Neither blocks nor log segments found. Creating new log.
I20260927 22:15:52.919657 32572 log.cc:824] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:15:52 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:52.922535 32572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: No bootstrap required, opened a new log
I20260927 22:15:52.930071 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:52.903234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.23.173.126:37215)
0927 22:15:52.904073 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:52.904084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:52.904126 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:15:52.905774 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:52.905795 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:52.906254 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:52.906588 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:52.908410 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:52.909735 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:52.911452 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:52.912081 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:52.912212 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:52.922733 (+ 10521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:52.922741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:52.922755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:52.922789 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:52.926030 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:52.926706 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:52.926728 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:52.926734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:52.926950 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:52.927331 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:52.927336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:52.927340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:52.927877 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:52.928207 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:52.928662 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 22:15:52.929071 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":368,"threads_started":1}
I20260927 22:15:52.930469 32572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:52.930893 32572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:15:52.930996 32572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af307a6f0e44c29bdc90fad59c62a6b, State: Initialized, Role: FOLLOWER
I20260927 22:15:52.931598 32572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:52.931828 32572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:52.931957 32572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:52.932183 32572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:15:52.934511 32572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:52.934970 32572 leader_election.cc:302] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af307a6f0e44c29bdc90fad59c62a6b; no voters:
I20260927 22:15:52.935506 32572 leader_election.cc:288] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:15:52.935649 32577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:15:52.936383 32577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 LEADER]: Becoming Leader. State: Replica: 9af307a6f0e44c29bdc90fad59c62a6b, State: Running, Role: LEADER
I20260927 22:15:52.936887 32577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:52.937492 32572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:52.939917 32578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } } }
I20260927 22:15:52.940335 32578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:52.941707 32579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af307a6f0e44c29bdc90fad59c62a6b. Latest consensus state: current_term: 1 leader_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } } }
I20260927 22:15:52.942099 32579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:52.942530 32585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:52.946642 32585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:52.954869 32585 catalog_manager.cc:1347] Generated new cluster ID: c4c7835a69e541c282d394443af0e560
I20260927 22:15:52.954963 32585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:52.973240 32585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:15:52.975047 32585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:52.988441 32585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Generated new TSK 0
I20260927 22:15:52.989281 32585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:15:53.051913 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:53.373908 32598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:53.374261 32598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:53.374435 32598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:53.381990 32598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:53.382156 32598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:53.382331 32598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:53.392154 32598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:53.393517 32598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:53.394860 32598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:53.407934 32606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:53.407896 32604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:53.407868 32603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:53.408803 32598 server_base.cc:1034] running on GCE node
I20260927 22:15:53.409380 32598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:53.410379 32598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:53.411612 32598 hybrid_clock.cc:648] HybridClock initialized: now 1790547353411575 us; error 46 us; skew 500 ppm
I20260927 22:15:53.411939 32598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:53 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547353, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:53.415091 32598 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:15:53.416555 32598 webserver.cc:460] Webserver started at http://127.23.173.65:34431/ using document root <none> and password file <none>
I20260927 22:15:53.417268 32598 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:53.417415 32598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:53.417724 32598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:53.420840 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "b532377242f544e4a3a2da17bb9defb3"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "665db22c7e1b81103fb6f041d4d09a15"
server_key_iv: "e80e550379f619d77a2c03590f52e770"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.421648 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "b532377242f544e4a3a2da17bb9defb3"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "665db22c7e1b81103fb6f041d4d09a15"
server_key_iv: "e80e550379f619d77a2c03590f52e770"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.426968 32598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:15:53.430550 32613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.431885 32598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:15:53.432065 32598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "b532377242f544e4a3a2da17bb9defb3"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "665db22c7e1b81103fb6f041d4d09a15"
server_key_iv: "e80e550379f619d77a2c03590f52e770"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.432328 32598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:53.447676 32598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:53.451646 32598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:53.451942 32598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:53.452944 32598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:53.454392 32598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:53.454495 32598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.454650 32598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:53.454725 32598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.488080 32598 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:41973
I20260927 22:15:53.488181 32725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:41973 every 8 connection(s)
I20260927 22:15:53.489461 32598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:53.493095 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 32598
I20260927 22:15:53.493321 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:15:53.494176 24245 external_mini_cluster.cc:1428] Setting key 4c7798065431ab3a159cda6bfefab03f
Sep 27 22:15:53 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547353, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:53.511606 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:53.492041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:60651 (local address 127.23.173.126:37215)
0927 22:15:53.492456 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:53.492463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:53.493756 (+ 1293us) server_negotiation.cc:407] Connection header received
0927 22:15:53.496021 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:53.496027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:53.496102 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:53.496221 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:53.498411 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.499126 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.501644 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.502137 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.502237 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:53.506065 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:53.506070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:53.506073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:53.506107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:53.508558 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:53.509275 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:53.509283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:53.509287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:53.509408 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:53.509849 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:53.509855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:53.509859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:53.510179 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:53.510332 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:53.511022 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 22:15:53.511240 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":191,"threads_started":1}
I20260927 22:15:53.512626 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:53.492651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:60651)
0927 22:15:53.493573 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:53.493625 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:53.495718 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:53.496520 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:53.496532 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:53.497326 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:53.498212 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:53.498252 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.499290 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.499297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:53.501416 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:53.501434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.502372 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.502381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:53.502710 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:53.503431 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:53.503471 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:53.505825 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:53.508699 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:53.508708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:53.508726 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:53.509141 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:53.509517 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:53.509521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:53.509525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:53.509706 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:53.510308 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:53.510316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:53.510655 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 22:15:53.510937 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:15:53.511470 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":303,"threads_started":1}
I20260927 22:15:53.514438 32726 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37215
I20260927 22:15:53.514745 32726 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:53.515439 32726 heartbeater.cc:511] Master 127.23.173.126:37215 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:15:53.517678 32537 ts_manager.cc:194] Registered new tserver with Master: b532377242f544e4a3a2da17bb9defb3 (127.23.173.65:41973)
I20260927 22:15:53.519119 32537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:60651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:15:53.544786 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:53.863442 32732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:53.863759 32732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:53.863934 32732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:53.871529 32732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:53.871676 32732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:53.871846 32732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:53.881598 32732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:53.882959 32732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:53.884248 32732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:53.897194 32738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:53.897166 32740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:53.898000 32737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:53.898168 32732 server_base.cc:1034] running on GCE node
I20260927 22:15:53.898829 32732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:53.899761 32732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:53.900988 32732 hybrid_clock.cc:648] HybridClock initialized: now 1790547353900929 us; error 52 us; skew 500 ppm
I20260927 22:15:53.901307 32732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:53 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547353, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:53.904495 32732 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:15:53.912933 32732 webserver.cc:460] Webserver started at http://127.23.173.66:40529/ using document root <none> and password file <none>
I20260927 22:15:53.913638 32732 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:53.913774 32732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:53.914144 32732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:53.917024 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "6e033520d429411eab7de542e2e711db"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "dfce49a959979dc4b8b384f98e2ba293"
server_key_iv: "f1c29c8f5be719f31b077cd42183f132"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.917804 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "6e033520d429411eab7de542e2e711db"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "dfce49a959979dc4b8b384f98e2ba293"
server_key_iv: "f1c29c8f5be719f31b077cd42183f132"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.922989 32732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:15:53.926462 32747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.927732 32732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:15:53.927906 32732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "6e033520d429411eab7de542e2e711db"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "dfce49a959979dc4b8b384f98e2ba293"
server_key_iv: "f1c29c8f5be719f31b077cd42183f132"
server_key_version: "encryptionkey@0"
I20260927 22:15:53.928097 32732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:53.946012 32732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:53.949851 32732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:53.950157 32732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:53.951090 32732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:53.952399 32732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:53.952471 32732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.952642 32732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:53.952692 32732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:53.985409 32732 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:35839
I20260927 22:15:53.985448 391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:35839 every 8 connection(s)
I20260927 22:15:53.986743 32732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:53.996390 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 32732
I20260927 22:15:53.996573 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:15:53.997133 24245 external_mini_cluster.cc:1428] Setting key f5e4638373bdb7ee9299aed3a40188b9
Sep 27 22:15:53 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547353, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:54.006956 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:53.989267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:58829 (local address 127.23.173.126:37215)
0927 22:15:53.989429 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:53.989434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:53.990621 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 22:15:53.992328 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:53.992333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:53.992398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:53.992484 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:53.993874 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.994632 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.996155 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.996852 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.996977 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.000318 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:54.000327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:54.000335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:54.000381 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:54.003927 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.004613 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.004621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.004627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.004790 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.005167 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.005174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.005180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.005541 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:54.005697 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:54.006386 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 22:15:54.006654 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:15:54.008157 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:53.989748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:58829)
0927 22:15:53.990471 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:53.990506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:53.992090 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:53.992692 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:53.992699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:53.993195 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:53.993736 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:53.993753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.994785 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.994791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:53.996015 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:53.996027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:53.997042 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:53.997047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:53.997268 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:53.997950 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:53.998036 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:54.000103 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:54.004066 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:54.004072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:54.004084 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:54.004495 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:54.004895 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:54.004898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:54.004900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:54.005065 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:54.005656 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:54.005662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:54.006018 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 22:15:54.006277 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:15:54.006649 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":306,"threads_started":1}
I20260927 22:15:54.010205 392 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37215
I20260927 22:15:54.010675 392 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:54.011538 392 heartbeater.cc:511] Master 127.23.173.126:37215 requested a full tablet report, sending...
I20260927 22:15:54.013120 32537 ts_manager.cc:194] Registered new tserver with Master: 6e033520d429411eab7de542e2e711db (127.23.173.66:35839)
I20260927 22:15:54.013939 32537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:58829
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:15:54.048660 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:54.377418 397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:54.377737 397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:54.377934 397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:54.385630 397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:54.385757 397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:54.385936 397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:54.396136 397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:54.398008 397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:54.399489 397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:54.415836 402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:54.415861 405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:54.416263 403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:54.416795 397 server_base.cc:1034] running on GCE node
I20260927 22:15:54.417376 397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:54.418421 397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:54.419646 397 hybrid_clock.cc:648] HybridClock initialized: now 1790547354419603 us; error 31 us; skew 500 ppm
I20260927 22:15:54.419974 397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:54 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547354, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:54.423292 397 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:15:54.424837 397 webserver.cc:460] Webserver started at http://127.23.173.67:45061/ using document root <none> and password file <none>
I20260927 22:15:54.425587 397 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:54.425734 397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:54.426131 397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:15:54.429018 397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "4ebe1183f6dc45f2aa9c721727049120"
format_stamp: "Formatted at 2026-09-27 22:15:54 on dist-test-slave-3wc3"
server_key: "e755580d33b49b036252c8b61f4a3ecd"
server_key_iv: "e4516577bd40c99baf940d8c7c190a99"
server_key_version: "encryptionkey@0"
I20260927 22:15:54.429778 397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebe1183f6dc45f2aa9c721727049120"
format_stamp: "Formatted at 2026-09-27 22:15:54 on dist-test-slave-3wc3"
server_key: "e755580d33b49b036252c8b61f4a3ecd"
server_key_iv: "e4516577bd40c99baf940d8c7c190a99"
server_key_version: "encryptionkey@0"
I20260927 22:15:54.434904 397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:15:54.438894 412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:54.440258 397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:15:54.440448 397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "4ebe1183f6dc45f2aa9c721727049120"
format_stamp: "Formatted at 2026-09-27 22:15:54 on dist-test-slave-3wc3"
server_key: "e755580d33b49b036252c8b61f4a3ecd"
server_key_iv: "e4516577bd40c99baf940d8c7c190a99"
server_key_version: "encryptionkey@0"
I20260927 22:15:54.440662 397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:54.456213 397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:54.460292 397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:54.460600 397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:54.461591 397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:54.462980 397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:54.463096 397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:54.463271 397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:54.463342 397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:54.496878 397 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:34033
I20260927 22:15:54.496922 524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:34033 every 8 connection(s)
I20260927 22:15:54.498246 397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:54.502657 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 397
I20260927 22:15:54.502856 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:15:54.503393 24245 external_mini_cluster.cc:1428] Setting key cd7f7227199eb1294878e29c356014e7
Sep 27 22:15:54 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547354, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:15:54.519748 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:54.500776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:32961 (local address 127.23.173.126:37215)
0927 22:15:54.500958 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:54.500963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:54.502162 (+ 1199us) server_negotiation.cc:407] Connection header received
0927 22:15:54.504723 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:54.504729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:54.504836 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:54.504920 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:54.506857 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.507625 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.509734 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.510221 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.510327 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.513750 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:54.513754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:54.513757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:54.513786 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:54.516315 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.517182 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.517188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.517193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.517357 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.517889 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.517894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.517898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.518288 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:54.518434 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:54.519308 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 22:15:54.519471 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:15:54.520761 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:54.501256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:32961)
0927 22:15:54.501922 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:54.502013 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:54.504413 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:54.505231 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:54.505240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:54.505842 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:54.506707 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:54.506726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.507766 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.507769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:54.509609 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:54.509620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.510430 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.510436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:54.510648 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.511359 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:54.511383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:54.513527 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:15:54.516444 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:54.516451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:54.516463 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:54.517002 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:54.517482 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:15:54.517487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:15:54.517490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:15:54.517772 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:15:54.518431 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:15:54.518439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:15:54.518855 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 22:15:54.519198 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 22:15:54.519590 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260927 22:15:54.522197 525 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37215
I20260927 22:15:54.522506 525 heartbeater.cc:464] Registering TS with master...
I20260927 22:15:54.522887 32726 heartbeater.cc:503] Master 127.23.173.126:37215 was elected leader, sending a full tablet report...
I20260927 22:15:54.523319 525 heartbeater.cc:511] Master 127.23.173.126:37215 requested a full tablet report, sending...
I20260927 22:15:54.524699 32537 ts_manager.cc:194] Registered new tserver with Master: 4ebe1183f6dc45f2aa9c721727049120 (127.23.173.67:34033)
I20260927 22:15:54.525528 32537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:32961
I20260927 22:15:54.531183 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
Entry for principal oryx/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab.
WARNING: no policy specified for oryx/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
Entry for principal oryx/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab.
WARNING: no policy specified for oryx/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
Entry for principal oryx/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab.
I20260927 22:15:54.668408 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 32598
Sep 27 22:15:54 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:15:54.684315 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:54.669626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.23.173.65:41973)
0927 22:15:54.670458 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:54.670466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:54.670494 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:15:54.670555 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:54.670561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:54.670782 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:54.670913 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:54.672209 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.673245 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.674533 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.675101 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.675206 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.679206 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:54.679217 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:54.679234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:54.679269 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:54.681889 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.682567 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.682574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.682576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.682696 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.682999 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.683001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.683003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.683328 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:54.683548 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:54.683816 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:15:54.684015 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":471,"threads_started":1}
I20260927 22:15:54.685644 32721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:54.769789 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 32598
I20260927 22:15:54.786352 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 32732
Sep 27 22:15:54 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:15:54.802721 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:54.787276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.23.173.66:35839)
0927 22:15:54.787846 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:54.787853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:54.787878 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:15:54.787947 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:54.787954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:54.788159 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:54.788336 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:54.789590 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.790711 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.791968 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.792527 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.792669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.797183 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:54.797198 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:54.797213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:54.797253 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:54.800186 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.800793 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.800800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.800802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.800921 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.801220 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.801223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.801225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.801539 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:54.801762 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:54.802097 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:15:54.802301 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
I20260927 22:15:54.804008 387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:54.886824 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 32732
I20260927 22:15:54.902801 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 397
Sep 27 22:15:54 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:15:54.919133 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:54.903880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.23.173.67:34033)
0927 22:15:54.904584 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:54.904592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:54.904620 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:15:54.904679 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:54.904685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:54.904901 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:54.905091 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:54.906518 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.907564 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.908844 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:54.909409 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:54.909562 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:54.913896 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:54.913907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:54.913922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:54.914014 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:54.916570 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.917196 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.917204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.917206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.917317 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:54.917633 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:54.917636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:54.917638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:54.918111 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:54.918389 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:54.918645 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:15:54.918851 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":370,"threads_started":1}
I20260927 22:15:54.920454 520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:55.003780 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 397
I20260927 22:15:55.020565 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 32505
I20260927 22:15:55.021703 32567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:15:55.124820 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 32505
I20260927 22:15:55.145498 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37215
--webserver_interface=127.23.173.126
--webserver_port=44339
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:15:55.470399 542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:55.470703 542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:55.470793 542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:55.478116 542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:15:55.478224 542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:55.478271 542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:55.478327 542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:15:55.478371 542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:15:55.488171 542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=44339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:55.489451 542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:55.490698 542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:55.502841 550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:55.502893 548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:55.502921 547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:55.503604 542 server_base.cc:1034] running on GCE node
I20260927 22:15:55.504350 542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:55.505716 542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:55.506910 542 hybrid_clock.cc:648] HybridClock initialized: now 1790547355506861 us; error 44 us; skew 500 ppm
I20260927 22:15:55.507268 542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:55 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547355, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:55.510368 542 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:15:55.511825 542 webserver.cc:460] Webserver started at http://127.23.173.126:44339/ using document root <none> and password file <none>
I20260927 22:15:55.512518 542 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:55.512670 542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:55.517860 542 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:55.520563 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:55.521822 542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:15:55.522083 542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9af307a6f0e44c29bdc90fad59c62a6b"
format_stamp: "Formatted at 2026-09-27 22:15:52 on dist-test-slave-3wc3"
server_key: "2fb40a12556d4ba20aefaef5ce9f0d57"
server_key_iv: "afe82446c3792fa945c3b77ae6cc6d39"
server_key_version: "encryptionkey@0"
I20260927 22:15:55.523063 542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:55.560261 542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:55.564388 542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:55.564683 542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:55.583120 542 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37215
I20260927 22:15:55.583141 608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37215 every 8 connection(s)
I20260927 22:15:55.584652 542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:15:55.588045 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 542
I20260927 22:15:55.588850 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:41973
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:37215
--webserver_port=34431
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:15:55.591635 609 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:15:55.594875 609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Bootstrap starting.
I20260927 22:15:55.623142 609 log.cc:824] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:15:55.634402 609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:15:55.635023 609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Bootstrap complete.
I20260927 22:15:55.648644 609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:55.649871 609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af307a6f0e44c29bdc90fad59c62a6b, State: Initialized, Role: FOLLOWER
I20260927 22:15:55.650596 609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:55.650872 609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:15:55.650993 609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:15:55.651165 609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:15:55.657549 609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:55.658227 609 leader_election.cc:302] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af307a6f0e44c29bdc90fad59c62a6b; no voters:
I20260927 22:15:55.658874 609 leader_election.cc:288] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:15:55.659027 613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:15:55.660195 613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [term 2 LEADER]: Becoming Leader. State: Replica: 9af307a6f0e44c29bdc90fad59c62a6b, State: Running, Role: LEADER
I20260927 22:15:55.660759 613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } }
I20260927 22:15:55.660960 609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:15:55.663456 615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af307a6f0e44c29bdc90fad59c62a6b. Latest consensus state: current_term: 2 leader_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } } }
I20260927 22:15:55.663787 615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:55.663429 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af307a6f0e44c29bdc90fad59c62a6b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37215 } } }
I20260927 22:15:55.664371 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b [sys.catalog]: This master's current role is: LEADER
I20260927 22:15:55.672672 618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:15:55.678414 618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:15:55.680049 618 catalog_manager.cc:1259] Loaded cluster ID: c4c7835a69e541c282d394443af0e560
I20260927 22:15:55.680213 618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:15:55.684302 618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:15:55.685822 618 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9af307a6f0e44c29bdc90fad59c62a6b: Loaded TSK: 0
I20260927 22:15:55.687408 618 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:15:55.957051 611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:55.957329 611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:55.957470 611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:55.964948 611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:55.965052 611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:55.965247 611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:15:55.975533 611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.65.keytab
--principal=oryx/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:41973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=34431
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:55.977005 611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:55.978443 611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:55.991291 635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:55.991300 636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:55.991921 638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:55.992610 611 server_base.cc:1034] running on GCE node
I20260927 22:15:55.993304 611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:55.994431 611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:55.995675 611 hybrid_clock.cc:648] HybridClock initialized: now 1790547355995613 us; error 47 us; skew 500 ppm
I20260927 22:15:55.996027 611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:55 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547355, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:55.999820 611 init.cc:377] Logged in from keytab as oryx/127.23.173.65@KRBTEST.COM (short username oryx)
I20260927 22:15:56.001549 611 webserver.cc:460] Webserver started at http://127.23.173.65:34431/ using document root <none> and password file <none>
I20260927 22:15:56.002473 611 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:56.002625 611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:56.009624 611 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 22:15:56.013144 645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.014812 611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:15:56.014982 611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "b532377242f544e4a3a2da17bb9defb3"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "665db22c7e1b81103fb6f041d4d09a15"
server_key_iv: "e80e550379f619d77a2c03590f52e770"
server_key_version: "encryptionkey@0"
I20260927 22:15:56.016337 611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:56.034456 611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:56.039184 611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:56.039467 611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:56.040534 611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:56.042194 611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:56.042304 611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.042384 611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:56.042439 611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.077714 611 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:41973
I20260927 22:15:56.077816 757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:41973 every 8 connection(s)
I20260927 22:15:56.079200 611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:15:56.088611 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 611
I20260927 22:15:56.089226 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:35839
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:37215
--webserver_port=40529
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:56.096609 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.082276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:53955)
0927 22:15:56.083080 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.083120 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.084713 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.085933 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.085941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.086494 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.087002 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.087018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.088575 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.088579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.089906 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.089918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.090964 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.090971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.091245 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.091915 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.091937 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.095750 (+ 3813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":382,"threads_started":1}
W20260927 22:15:56.097265 758 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37215 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.126:37215: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
W20260927 22:15:56.097314 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.082164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:53955 (local address 127.23.173.126:37215)
0927 22:15:56.082940 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.082956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.083255 (+ 299us) server_negotiation.cc:407] Connection header received
0927 22:15:56.085197 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.085209 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.085517 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.085796 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.087161 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.088409 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.090093 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.090736 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.090934 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.096173 (+ 5239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:53955: BlockingRecv error: recv got EOF from 127.23.173.65:53955 (error 108)
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":260,"threads_started":1}
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:56.105482 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.097892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:57257)
0927 22:15:56.098126 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.098140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.098273 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.098598 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.098602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.098931 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.099057 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.099065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.100037 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.100043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.100939 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.100949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.101621 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.101627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.101798 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.102391 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.102408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.105207 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 22:15:56.105880 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.098160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:57257 (local address 127.23.173.126:37215)
0927 22:15:56.098341 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.098347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.098365 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:15:56.098410 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.098415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.098475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.098574 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.099190 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.099907 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.101081 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.101478 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.101600 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.105401 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:57257: BlockingRecv error: recv got EOF from 127.23.173.65:57257 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:56.113571 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:58347)
0927 22:15:56.106288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.106303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.106374 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.106774 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.106778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.107071 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.107206 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.107214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.108185 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.108190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.109161 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.109172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.109843 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.109848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.110060 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.110661 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.110677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.113281 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:15:56.113848 758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:15:56.114014 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.106356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:58347 (local address 127.23.173.126:37215)
0927 22:15:56.106533 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.106537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.106549 (+ 12us) server_negotiation.cc:407] Connection header received
0927 22:15:56.106584 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.106587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.106648 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.106755 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.107338 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.108048 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.109297 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.109694 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.109822 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.113471 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:58347: BlockingRecv error: recv got EOF from 127.23.173.65:58347 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:15:56.409255 762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:56.409576 762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:56.409750 762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:56.417032 762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:56.417193 762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:56.417385 762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:15:56.427649 762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.66.keytab
--principal=oryx/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:35839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=40529
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:56.429005 762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:56.430415 762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:56.443253 767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:56.443337 768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:56.443280 770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:56.444329 762 server_base.cc:1034] running on GCE node
I20260927 22:15:56.444924 762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:56.445818 762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:56.447041 762 hybrid_clock.cc:648] HybridClock initialized: now 1790547356446992 us; error 63 us; skew 500 ppm
I20260927 22:15:56.447352 762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547356, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:56.450529 762 init.cc:377] Logged in from keytab as oryx/127.23.173.66@KRBTEST.COM (short username oryx)
I20260927 22:15:56.451962 762 webserver.cc:460] Webserver started at http://127.23.173.66:40529/ using document root <none> and password file <none>
I20260927 22:15:56.452593 762 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:56.452702 762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:56.457888 762 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:56.460471 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.461874 762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:15:56.462100 762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "6e033520d429411eab7de542e2e711db"
format_stamp: "Formatted at 2026-09-27 22:15:53 on dist-test-slave-3wc3"
server_key: "dfce49a959979dc4b8b384f98e2ba293"
server_key_iv: "f1c29c8f5be719f31b077cd42183f132"
server_key_version: "encryptionkey@0"
I20260927 22:15:56.463037 762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:56.502799 762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:56.507184 762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:56.507463 762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:56.508431 762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:56.509742 762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:56.509868 762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.510025 762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:56.510120 762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.542999 762 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:35839
I20260927 22:15:56.543092 889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:35839 every 8 connection(s)
I20260927 22:15:56.544484 762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:15:56.552731 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 762
I20260927 22:15:56.553318 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:34033
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:37215
--webserver_port=45061
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:15:56.560146 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.547093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:55731 (local address 127.23.173.126:37215)
0927 22:15:56.547274 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.547279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.548396 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 22:15:56.550280 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.550287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.550381 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.550493 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.551945 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.552670 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.554380 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.554804 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.554935 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.559869 (+ 4934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:55731: BlockingRecv error: recv got EOF from 127.23.173.66:55731 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:15:56.560664 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.547496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:55731)
0927 22:15:56.548237 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.548273 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.549956 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.550726 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.550734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.551219 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.551769 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.551786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.552837 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.552842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.554239 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.554251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.555033 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.555042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.555276 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.555949 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.555970 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.559600 (+ 3630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":268,"threads_started":1}
W20260927 22:15:56.561280 890 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37215 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.126:37215: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:56.569530 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.561865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:33357)
0927 22:15:56.562066 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.562081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.562232 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.562530 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.562534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.562867 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.562996 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.563004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.563954 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.563959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.564834 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.564844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.565524 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.565529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.565693 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.566283 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.566299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.569252 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:15:56.569681 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.562041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:33357 (local address 127.23.173.126:37215)
0927 22:15:56.562248 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.562253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.562273 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:15:56.562340 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.562345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.562408 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.562508 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.563131 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.563820 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.564974 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.565375 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.565514 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.569444 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:33357: BlockingRecv error: recv got EOF from 127.23.173.66:33357 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:56.577708 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.570269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:44383)
0927 22:15:56.570401 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.570419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.570518 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.570829 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.570834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.571136 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.571273 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.571281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.572232 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.572237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.573203 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.573214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.574018 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.574023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.574239 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.574840 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:56.574856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:56.577425 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 22:15:56.578070 890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:15:56.578204 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.570379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:44383 (local address 127.23.173.126:37215)
0927 22:15:56.570549 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.570557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.570588 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:15:56.570637 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.570642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.570706 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.570806 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.571406 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.572097 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.573345 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.573861 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.574004 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:56.577613 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:44383: BlockingRecv error: recv got EOF from 127.23.173.66:44383 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 22:15:56.876932 893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:15:56.877327 893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:15:56.877503 893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:15:56.884894 893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:15:56.885048 893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:15:56.885242 893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:15:56.894990 893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/krb5kdc/oryx_127.23.173.67.keytab
--principal=oryx/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:34033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=45061
--tserver_master_addrs=127.23.173.126:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:15:56.896350 893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:15:56.897704 893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:15:56.910943 898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:56.910948 899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:15:56.910975 901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:15:56.912029 893 server_base.cc:1034] running on GCE node
I20260927 22:15:56.912688 893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:15:56.913683 893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:15:56.914908 893 hybrid_clock.cc:648] HybridClock initialized: now 1790547356914839 us; error 66 us; skew 500 ppm
I20260927 22:15:56.915285 893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:15:56 dist-test-slave-3wc3 krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547356, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:15:56.918360 893 init.cc:377] Logged in from keytab as oryx/127.23.173.67@KRBTEST.COM (short username oryx)
I20260927 22:15:56.919793 893 webserver.cc:460] Webserver started at http://127.23.173.67:45061/ using document root <none> and password file <none>
I20260927 22:15:56.920490 893 fs_manager.cc:359] Metadata directory not provided
I20260927 22:15:56.920715 893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:15:56.926607 893 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:15:56.929349 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.930660 893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:15:56.930850 893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "4ebe1183f6dc45f2aa9c721727049120"
format_stamp: "Formatted at 2026-09-27 22:15:54 on dist-test-slave-3wc3"
server_key: "e755580d33b49b036252c8b61f4a3ecd"
server_key_iv: "e4516577bd40c99baf940d8c7c190a99"
server_key_version: "encryptionkey@0"
I20260927 22:15:56.931824 893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:15:56.946669 893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:15:56.950555 893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:15:56.950773 893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:15:56.951637 893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:15:56.952848 893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:15:56.952931 893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.953008 893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:15:56.953064 893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:15:56.987481 893 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:34033
I20260927 22:15:56.987593 1020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:34033 every 8 connection(s)
I20260927 22:15:56.988970 893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:15:56.996371 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 893
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:15:57.005750 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:56.991427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:58317 (local address 127.23.173.126:37215)
0927 22:15:56.991661 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.991667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.992803 (+ 1136us) server_negotiation.cc:407] Connection header received
0927 22:15:56.994626 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.994630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.994700 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.994792 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:56.996228 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.997218 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.999634 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.000040 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.000133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.005455 (+ 5322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:58317: BlockingRecv error: recv got EOF from 127.23.173.67:58317 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:15:57.006244 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.991946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:58317)
0927 22:15:56.992628 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:56.992665 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:56.994383 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:56.994993 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:56.995001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:56.995493 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:56.996064 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.996080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:56.997384 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:56.997388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:56.999497 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:56.999513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.000277 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.000283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.000477 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.001138 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:57.001161 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:57.005167 (+ 4006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":273,"threads_started":1}
W20260927 22:15:57.007031 1021 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37215 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.173.126:37215: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:57.013720 1025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:56.997420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.23.173.126:37215)
0927 22:15:56.998258 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:56.998261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:56.998275 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:15:56.998310 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:56.998313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:56.998362 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:56.998442 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:57.000271 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.001164 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.002969 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.003435 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.003566 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.006454 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:15:57.006465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:15:57.006486 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:15:57.006529 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:15:57.010476 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:57.011150 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:57.011159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:57.011174 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:57.011393 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:15:57.011774 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:15:57.011779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:15:57.011782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:15:57.012355 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:15:57.012649 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:15:57.013052 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 22:15:57.013403 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":632,"threads_started":1}
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:57.015261 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:57.007643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:36615)
0927 22:15:57.007832 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:57.007847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:57.007934 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:57.008198 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:57.008201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:57.008470 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:57.008614 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.008621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.010050 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.010053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.010917 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.010926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.011653 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.011658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.011820 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.012274 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:57.012290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:57.015072 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:15:57.015537 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:57.007745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:36615 (local address 127.23.173.126:37215)
0927 22:15:57.007929 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:57.007938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:57.007957 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:15:57.008009 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:57.008014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:57.008078 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:57.008175 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:57.008735 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.009889 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.011053 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.011514 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.011642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.015313 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:36615: BlockingRecv error: recv got EOF from 127.23.173.67:36615 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:57.024322 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:57.016196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:45505)
0927 22:15:57.016433 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:57.016451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:57.016541 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:57.016874 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:57.016878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:57.017197 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:57.017379 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.017394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.018411 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.018415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.019293 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.019303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.019904 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.019907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.020158 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.020840 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:57.020864 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:57.024114 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260927 22:15:57.024487 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:57.016262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:45505 (local address 127.23.173.126:37215)
0927 22:15:57.016471 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:57.016476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:57.016499 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:15:57.016642 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:57.016645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:57.016738 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:57.016832 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:57.017516 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.018287 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.019401 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.019775 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.019883 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.024290 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:45505: BlockingRecv error: recv got EOF from 127.23.173.67:45505 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260927 22:15:57.024638 1021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:57.122551 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:57.114410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:54033)
0927 22:15:57.114945 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:57.114965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:57.115078 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:57.115358 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:57.115361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:57.115810 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:57.115999 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.116008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.116930 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.116934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.117878 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.117889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.118597 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.118601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.118727 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.119339 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:57.119356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:57.122262 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
W20260927 22:15:57.122790 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:57.114474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:54033 (local address 127.23.173.126:37215)
0927 22:15:57.114635 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:57.114640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:57.115035 (+ 395us) server_negotiation.cc:407] Connection header received
0927 22:15:57.115173 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:57.115177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:57.115235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:57.115312 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:57.116118 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.116813 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.118040 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.118453 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.118541 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.122533 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:54033: BlockingRecv error: recv got EOF from 127.23.173.65:54033 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:57 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:57.587275 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:57.578596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:40099)
0927 22:15:57.579092 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:57.579106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:57.579268 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:57.579599 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:57.579603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:57.580084 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:57.580286 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.580296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.581285 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.581290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.582350 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:57.582361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.583191 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.583197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:57.583375 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.584009 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:57.584026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:57.586949 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
W20260927 22:15:57.587456 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:57.578676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:40099 (local address 127.23.173.126:37215)
0927 22:15:57.578902 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:57.578909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:57.579187 (+ 278us) server_negotiation.cc:407] Connection header received
0927 22:15:57.579389 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:57.579395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:57.579463 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:57.579565 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:57.580427 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.581127 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.582503 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:57.583009 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:57.583159 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:57.587182 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:40099: BlockingRecv error: recv got EOF from 127.23.173.66:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:58.041882 1028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:58.025291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:40477)
0927 22:15:58.025840 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:58.025859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:58.026051 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:58.026468 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:58.026473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:58.027053 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:58.027255 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.027264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.028276 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.028280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.029377 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.029389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.030062 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.030067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.030180 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.038611 (+ 8431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:58.038630 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:58.041536 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
W20260927 22:15:58.041992 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:58.025346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:40477 (local address 127.23.173.126:37215)
0927 22:15:58.025585 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:58.025593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:58.026006 (+ 413us) server_negotiation.cc:407] Connection header received
0927 22:15:58.026209 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:58.026216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:58.026307 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:58.026412 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:58.027409 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.028140 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.029493 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.029867 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.029999 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.041744 (+ 11745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:40477: BlockingRecv error: recv got EOF from 127.23.173.67:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:58.132189 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:58.123487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:58455)
0927 22:15:58.124000 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:58.124019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:58.124175 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:58.124510 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:58.124515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:58.124922 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:58.125086 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.125095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.126165 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.126172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.127236 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.127254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.127999 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.128004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.128165 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.128698 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:58.128714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:58.131823 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
W20260927 22:15:58.132333 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:58.123539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:58455 (local address 127.23.173.126:37215)
0927 22:15:58.123779 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:58.123788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:58.124102 (+ 314us) server_negotiation.cc:407] Connection header received
0927 22:15:58.124320 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:58.124324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:58.124385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:58.124484 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:58.125227 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.126038 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.127380 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.127829 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.127986 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.132042 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:58455: BlockingRecv error: recv got EOF from 127.23.173.65:58455 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:58 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:58.596696 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:58.588580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:50545)
0927 22:15:58.589152 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:58.589168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:58.589276 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:58.589563 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:58.589566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:58.589991 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:58.590163 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.590172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.591131 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.591135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.592174 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:58.592186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.592976 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.592983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:58.593111 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.593753 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:58.593770 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:58.596459 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":328,"threads_started":1}
W20260927 22:15:58.596837 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:58.588652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50545 (local address 127.23.173.126:37215)
0927 22:15:58.588824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:58.588829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:58.589251 (+ 422us) server_negotiation.cc:407] Connection header received
0927 22:15:58.589371 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:58.589374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:58.589433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:58.589519 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:58.590285 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.591009 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.592309 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:58.592843 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:58.592933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:58.596642 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:50545: BlockingRecv error: recv got EOF from 127.23.173.66:50545 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:59.051380 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:59.042868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:43851)
0927 22:15:59.043427 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:59.043443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:59.043557 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:59.043855 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:59.043859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:59.044237 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:59.044404 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.044425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.045375 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.045378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.046346 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.046358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.047201 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.047205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.047359 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.048093 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:59.048118 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:59.051019 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":312,"threads_started":1}
W20260927 22:15:59.051466 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:59.043029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:43851 (local address 127.23.173.126:37215)
0927 22:15:59.043215 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:59.043219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:59.043520 (+ 301us) server_negotiation.cc:407] Connection header received
0927 22:15:59.043656 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:59.043659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:59.043720 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:59.043802 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:59.044533 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.045243 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.046464 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.047031 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.047179 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.051239 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:43851: BlockingRecv error: recv got EOF from 127.23.173.67:43851 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:59.140978 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:59.133185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:47967)
0927 22:15:59.133701 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:59.133734 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:59.133847 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:59.134156 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:59.134159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:59.134493 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:59.134649 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.134657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.135743 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.135746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.136679 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.136690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.137295 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.137299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.137421 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.138035 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:59.138054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:59.140726 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":282,"threads_started":1}
W20260927 22:15:59.141175 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:59.133228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:47967 (local address 127.23.173.126:37215)
0927 22:15:59.133427 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:59.133431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:59.133808 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:15:59.133944 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:59.133947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:59.134026 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:59.134109 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:59.134833 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.135609 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.136809 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.137179 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.137273 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.140926 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:47967: BlockingRecv error: recv got EOF from 127.23.173.65:47967 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:15:59 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:15:59.605515 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:15:59.597724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:37595)
0927 22:15:59.598241 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 22:15:59.598263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:15:59.598395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:15:59.598760 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:15:59.598768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:15:59.599113 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:15:59.599285 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.599293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.600285 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.600288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.601178 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:15:59.601189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.601803 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.601807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:15:59.601933 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.602509 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:15:59.602526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:15:59.605224 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":287,"threads_started":1}
W20260927 22:15:59.605628 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:15:59.597790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:37595 (local address 127.23.173.126:37215)
0927 22:15:59.598026 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:15:59.598034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:15:59.598335 (+ 301us) server_negotiation.cc:407] Connection header received
0927 22:15:59.598511 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:15:59.598515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:15:59.598599 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:15:59.598692 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:15:59.599417 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.600167 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.601290 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:15:59.601663 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:15:59.601784 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:15:59.605415 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:37595: BlockingRecv error: recv got EOF from 127.23.173.66:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:00.061616 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:00.052368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:46757)
0927 22:16:00.052928 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:00.052968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:00.053105 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:00.053486 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:00.053491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:00.053871 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:00.054102 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.054113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.055116 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.055121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.056109 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.056123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.057060 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.057065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.057294 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.058001 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:00.058027 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:00.061252 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
W20260927 22:16:00.061728 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:00.052450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:46757 (local address 127.23.173.126:37215)
0927 22:16:00.052646 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:00.052654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:00.053035 (+ 381us) server_negotiation.cc:407] Connection header received
0927 22:16:00.053261 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:00.053267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:00.053352 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:00.053452 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:00.054245 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.054979 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.056337 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.056901 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.057044 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.061487 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:46757: BlockingRecv error: recv got EOF from 127.23.173.67:46757 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:00.150485 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:00.142042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:35033)
0927 22:16:00.142574 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:00.142607 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:00.142719 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:00.143024 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:00.143027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:00.143397 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:00.143544 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.143552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.144494 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.144497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.145615 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.145628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.146467 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.146473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.146650 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.147281 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:00.147299 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:00.150145 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":302,"threads_started":1}
W20260927 22:16:00.150612 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:00.142134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:35033 (local address 127.23.173.126:37215)
0927 22:16:00.142373 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:00.142378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:00.142685 (+ 307us) server_negotiation.cc:407] Connection header received
0927 22:16:00.142821 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:00.142824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:00.142884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:00.142970 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:00.143664 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.144387 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.145765 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.146273 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.146444 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.150364 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:35033: BlockingRecv error: recv got EOF from 127.23.173.65:35033 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:00 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:00.614521 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:00.606512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:34347)
0927 22:16:00.607049 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:00.607064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:00.607178 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:00.607514 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:00.607517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:00.607898 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:00.608043 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.608050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.608976 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.608980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.609919 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:00.609930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.610564 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.610568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:00.610689 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.611235 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:00.611252 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:00.614294 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":312,"threads_started":1}
W20260927 22:16:00.614830 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:00.606564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34347 (local address 127.23.173.126:37215)
0927 22:16:00.606725 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:00.606730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:00.607135 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:16:00.607332 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:00.607336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:00.607395 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:00.607475 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:00.608167 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.608869 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.610055 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:00.610448 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:00.610536 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:00.614524 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:34347: BlockingRecv error: recv got EOF from 127.23.173.66:34347 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:01.071451 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:01.062648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:33397)
0927 22:16:01.063101 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:01.063117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:01.063240 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:01.063538 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:01.063541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:01.063913 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:01.064061 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.064070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.065293 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.065298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.066826 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.066841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.067534 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.067538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.067679 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.068334 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:01.068352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:01.071228 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":251,"threads_started":1}
W20260927 22:16:01.071740 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:01.062680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:33397 (local address 127.23.173.126:37215)
0927 22:16:01.062932 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:01.062938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:01.063204 (+ 266us) server_negotiation.cc:407] Connection header received
0927 22:16:01.063341 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:01.063345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:01.063412 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:01.063495 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:01.064215 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.065162 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.066967 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.067400 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.067514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.071440 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:33397: BlockingRecv error: recv got EOF from 127.23.173.67:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:01.160748 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:01.151768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:54987)
0927 22:16:01.152365 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:01.152385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:01.152544 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:01.152914 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:01.152919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:01.153432 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:01.153594 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.153603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.154706 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.154712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.155629 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.155642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.156378 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.156384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.156572 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.157318 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:01.157337 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:01.160384 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":316,"threads_started":1}
W20260927 22:16:01.160874 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:01.151868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:54987 (local address 127.23.173.126:37215)
0927 22:16:01.152136 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:01.152143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:01.152475 (+ 332us) server_negotiation.cc:407] Connection header received
0927 22:16:01.152706 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:01.152710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:01.152771 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:01.152875 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:01.153754 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.154559 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.155783 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.156225 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.156360 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.160615 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:54987: BlockingRecv error: recv got EOF from 127.23.173.65:54987 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:01 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:01.623615 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:01.615438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:50877)
0927 22:16:01.615969 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:01.615984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:01.616092 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:01.616388 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:01.616392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:01.616778 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:01.616953 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.616962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.617912 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.617915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.618867 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:01.618878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.619499 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.619502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:01.619622 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.620282 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:01.620310 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:01.623271 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
W20260927 22:16:01.623690 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:01.615571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50877 (local address 127.23.173.126:37215)
0927 22:16:01.615753 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:01.615757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:01.616058 (+ 301us) server_negotiation.cc:407] Connection header received
0927 22:16:01.616191 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:01.616195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:01.616269 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:01.616347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:01.617079 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.617797 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.618984 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:01.619383 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:01.619472 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:01.623463 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:50877: BlockingRecv error: recv got EOF from 127.23.173.66:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:02.080714 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:02.072443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:36401)
0927 22:16:02.072975 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:02.072991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:02.073137 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:02.073488 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:02.073493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:02.074018 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:02.074192 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.074200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.075175 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.075182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.076108 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.076118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.076832 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.076837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.077017 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.077562 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:02.077579 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:02.080368 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":238,"threads_started":1}
W20260927 22:16:02.080785 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:02.072537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:36401 (local address 127.23.173.126:37215)
0927 22:16:02.072751 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:02.072760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:02.073072 (+ 312us) server_negotiation.cc:407] Connection header received
0927 22:16:02.073287 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:02.073292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:02.073360 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:02.073459 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:02.074335 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.075039 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.076252 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.076681 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.076811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.080559 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:36401: BlockingRecv error: recv got EOF from 127.23.173.67:36401 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:02.170279 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:02.161753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:51047)
0927 22:16:02.162359 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:02.162381 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:02.162499 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:02.162794 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:02.162797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:02.163144 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:02.163318 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.163326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.164293 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.164296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.165175 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.165186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.166039 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.166045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.166254 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.167060 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:02.167086 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:02.169943 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
W20260927 22:16:02.170420 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:02.161810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:51047 (local address 127.23.173.126:37215)
0927 22:16:02.162045 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:02.162054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:02.162462 (+ 408us) server_negotiation.cc:407] Connection header received
0927 22:16:02.162600 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:02.162604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:02.162667 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:02.162751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:02.163435 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.164169 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.165328 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.165837 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.165930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.170158 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:51047: BlockingRecv error: recv got EOF from 127.23.173.65:51047 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:02 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:02.632925 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:02.624848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:50179)
0927 22:16:02.625415 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:02.625433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:02.625556 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:02.625970 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:02.625975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:02.626435 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:02.626593 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.626601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.627742 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.627746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.628626 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:02.628636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.629278 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.629282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:02.629399 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.630001 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:02.630028 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:02.632711 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":328,"threads_started":1}
W20260927 22:16:02.633227 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:02.624939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50179 (local address 127.23.173.126:37215)
0927 22:16:02.625134 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:02.625160 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:02.625541 (+ 381us) server_negotiation.cc:407] Connection header received
0927 22:16:02.625664 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:02.625669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:02.625768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:02.625891 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:02.626716 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.627625 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.628745 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:02.629144 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:02.629245 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:02.632954 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:50179: BlockingRecv error: recv got EOF from 127.23.173.66:50179 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:03.094635 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:03.081631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:32907)
0927 22:16:03.082240 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:03.082256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:03.082363 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:03.082628 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:03.082631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:03.083000 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:03.083163 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.083171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.084145 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.084149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.085054 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.085064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.085776 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.085780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.085902 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.091671 (+ 5769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:03.091690 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:03.094425 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
W20260927 22:16:03.094942 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:03.081738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:32907 (local address 127.23.173.126:37215)
0927 22:16:03.081902 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:03.081906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:03.082328 (+ 422us) server_negotiation.cc:407] Connection header received
0927 22:16:03.082445 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:03.082448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:03.082506 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:03.082584 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:03.083284 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.084027 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.085178 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.085605 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.085704 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.094667 (+ 8963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:32907: BlockingRecv error: recv got EOF from 127.23.173.67:32907 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:03.180020 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:03.171127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:52099)
0927 22:16:03.171714 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:03.171740 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:03.171852 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:03.172150 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:03.172153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:03.172640 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:03.172839 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.172849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.174266 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.174272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.175535 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.175552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.176409 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.176414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.176526 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.177030 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:03.177048 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:03.179769 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":275,"threads_started":1}
W20260927 22:16:03.180150 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:03.171260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:52099 (local address 127.23.173.126:37215)
0927 22:16:03.171448 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:03.171453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:03.171815 (+ 362us) server_negotiation.cc:407] Connection header received
0927 22:16:03.171953 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:03.171956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:03.172017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:03.172107 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:03.172983 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.174048 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.175677 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.176268 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.176361 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.179952 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:52099: BlockingRecv error: recv got EOF from 127.23.173.65:52099 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:03 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:03.641808 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:03.633832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:60599)
0927 22:16:03.634385 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:03.634400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:03.634510 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:03.634801 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:03.634804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:03.635194 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:03.635458 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.635469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.636500 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.636505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.637409 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:03.637421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.638091 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.638096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:03.638209 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.638769 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:03.638785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:03.641556 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":263,"threads_started":1}
W20260927 22:16:03.642015 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:03.633879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:60599 (local address 127.23.173.126:37215)
0927 22:16:03.634139 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:03.634148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:03.634477 (+ 329us) server_negotiation.cc:407] Connection header received
0927 22:16:03.634613 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:03.634617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:03.634678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:03.634757 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:03.635580 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.636326 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.637528 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:03.637927 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:03.638040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:03.641750 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:60599: BlockingRecv error: recv got EOF from 127.23.173.66:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:04.103968 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:04.095799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:53567)
0927 22:16:04.096414 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:04.096430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:04.096540 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:04.096831 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:04.096834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:04.097245 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:04.097419 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.097427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.098550 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.098558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.099466 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.099477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.100225 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.100230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.100404 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.101013 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:04.101031 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:04.103731 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":281,"threads_started":1}
W20260927 22:16:04.104141 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:04.095842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:53567 (local address 127.23.173.126:37215)
0927 22:16:04.096004 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:04.096009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:04.096503 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:16:04.096634 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:04.096638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:04.096700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:04.096786 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:04.097577 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.098407 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.099629 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.100070 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.100196 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.103925 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:53567: BlockingRecv error: recv got EOF from 127.23.173.67:53567 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:04.189145 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:04.181174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:44255)
0927 22:16:04.181754 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:04.181780 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:04.181914 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:04.182408 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:04.182414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:04.182821 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:04.182973 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.182982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.184022 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.184026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.184921 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.184931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.185576 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.185581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.185685 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.186289 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:04.186305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:04.188917 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":420,"threads_started":1}
W20260927 22:16:04.189479 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:04.181384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:44255 (local address 127.23.173.126:37215)
0927 22:16:04.181593 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:04.181600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:04.181865 (+ 265us) server_negotiation.cc:407] Connection header received
0927 22:16:04.182074 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:04.182082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:04.182222 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:04.182353 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:04.183120 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.183898 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.185037 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.185432 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.185520 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.189101 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:44255: BlockingRecv error: recv got EOF from 127.23.173.65:44255 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:04 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:04.651119 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:04.642930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:34637)
0927 22:16:04.643466 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:04.643483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:04.643595 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:04.643903 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:04.643907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:04.644312 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:04.644472 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.644480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.645578 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.645582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.646670 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:04.646707 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.647588 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.647592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:04.647714 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.648322 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:04.648338 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:04.650890 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":314,"threads_started":1}
W20260927 22:16:04.651286 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:04.642986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34637 (local address 127.23.173.126:37215)
0927 22:16:04.643230 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:04.643240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:04.643566 (+ 326us) server_negotiation.cc:407] Connection header received
0927 22:16:04.643699 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:04.643703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:04.643763 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:04.643846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:04.644592 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.645439 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.646831 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:04.647432 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:04.647556 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:04.651069 (+ 3513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:34637: BlockingRecv error: recv got EOF from 127.23.173.66:34637 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:05.113067 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:05.104797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:34807)
0927 22:16:05.105314 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:05.105328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:05.105478 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:05.105823 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:05.105827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:05.106230 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:05.106384 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.106393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.107405 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.107411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.108322 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.108332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.109096 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.109103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.109275 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.110007 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:05.110041 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:05.112837 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":262,"threads_started":1}
W20260927 22:16:05.113370 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:05.104872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:34807 (local address 127.23.173.126:37215)
0927 22:16:05.105121 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:05.105129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:05.105411 (+ 282us) server_negotiation.cc:407] Connection header received
0927 22:16:05.105601 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:05.105606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:05.105680 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:05.105778 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:05.106528 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.107249 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.108474 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.108918 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.109062 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.113083 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:34807: BlockingRecv error: recv got EOF from 127.23.173.67:34807 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:05.198868 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:05.190000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:49147)
0927 22:16:05.190579 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:05.190599 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:05.190759 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:05.191142 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:05.191147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:05.191566 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:05.191727 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.191736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.192817 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.192822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.193768 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.193787 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.194569 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.194575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.194760 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.195294 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:05.195311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:05.198513 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
W20260927 22:16:05.198937 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:05.190204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:49147 (local address 127.23.173.126:37215)
0927 22:16:05.190461 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:05.190467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:05.190687 (+ 220us) server_negotiation.cc:407] Connection header received
0927 22:16:05.190925 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:05.190930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:05.191006 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:05.191109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:05.191885 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.192656 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.193922 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.194416 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.194555 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.198710 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:49147: BlockingRecv error: recv got EOF from 127.23.173.65:49147 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:05 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:05.660313 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:05.652049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:36927)
0927 22:16:05.652548 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:05.652564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:05.652674 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:05.652975 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:05.652978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:05.653355 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:05.653516 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.653524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.654571 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.654574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.655547 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:05.655558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.656233 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.656236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:05.656355 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.656897 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:05.656914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:05.659996 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
W20260927 22:16:05.660521 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:05.652153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36927 (local address 127.23.173.126:37215)
0927 22:16:05.652405 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:05.652411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:05.652647 (+ 236us) server_negotiation.cc:407] Connection header received
0927 22:16:05.652785 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:05.652789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:05.652851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:05.652929 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:05.653632 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.654458 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.655674 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:05.656110 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:05.656200 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:05.660258 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:36927: BlockingRecv error: recv got EOF from 127.23.173.66:36927 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:06.122382 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:06.114194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:51031)
0927 22:16:06.114774 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:06.114796 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:06.114905 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:06.115173 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:06.115176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:06.115564 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:06.115741 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.115750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.116680 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.116684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.117610 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.117620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.118418 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.118424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.118600 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.119135 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:06.119153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:06.122031 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":318,"threads_started":1}
W20260927 22:16:06.122413 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:06.114282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:51031 (local address 127.23.173.126:37215)
0927 22:16:06.114472 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:06.114476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:06.114853 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:16:06.114986 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:06.114990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:06.115050 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:06.115125 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:06.115885 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.116569 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.117778 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.118253 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.118401 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.122221 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:51031: BlockingRecv error: recv got EOF from 127.23.173.67:51031 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:06.207569 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:06.199884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:40521)
0927 22:16:06.200343 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:06.200363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:06.200474 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:06.200767 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:06.200770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:06.201127 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:06.201267 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.201275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.202317 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.202321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.203241 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.203252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.203936 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.203940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.204058 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.204630 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:06.204647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:06.207348 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":263,"threads_started":1}
W20260927 22:16:06.207757 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:06.199942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:40521 (local address 127.23.173.126:37215)
0927 22:16:06.200164 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:06.200173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:06.200446 (+ 273us) server_negotiation.cc:407] Connection header received
0927 22:16:06.200585 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:06.200588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:06.200647 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:06.200724 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:06.201397 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.202183 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.203369 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.203794 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.203885 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.207514 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:40521: BlockingRecv error: recv got EOF from 127.23.173.65:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:06 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:06.669162 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:06.661304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:36071)
0927 22:16:06.661869 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:06.661895 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:06.662056 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:06.662367 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:06.662370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:06.662714 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:06.662870 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.662878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.663790 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.663793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.664717 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:06.664727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.665388 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.665392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:06.665500 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.666062 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:06.666079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:06.668929 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":320,"threads_started":1}
W20260927 22:16:06.669353 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:06.661319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36071 (local address 127.23.173.126:37215)
0927 22:16:06.661477 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:06.661481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:06.662014 (+ 533us) server_negotiation.cc:407] Connection header received
0927 22:16:06.662180 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:06.662184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:06.662246 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:06.662326 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:06.662982 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.663678 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.664843 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:06.665267 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:06.665358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:06.669103 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:36071: BlockingRecv error: recv got EOF from 127.23.173.66:36071 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:07.131927 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:07.123424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:45993)
0927 22:16:07.124040 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:07.124059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:07.124188 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:07.124475 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:07.124478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:07.124836 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:07.124980 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.124988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.126029 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.126033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.127042 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.127055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.127779 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.127783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.127958 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.128519 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:07.128536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:07.131697 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":343,"threads_started":1}
W20260927 22:16:07.132143 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:07.123559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:45993 (local address 127.23.173.126:37215)
0927 22:16:07.123834 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:07.123841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:07.124151 (+ 310us) server_negotiation.cc:407] Connection header received
0927 22:16:07.124289 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:07.124292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:07.124351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:07.124432 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:07.125094 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.125844 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.127169 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.127657 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.127747 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.131883 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:45993: BlockingRecv error: recv got EOF from 127.23.173.67:45993 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:07.216758 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:07.208468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:38723)
0927 22:16:07.209005 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:07.209025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:07.209133 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:07.209475 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:07.209478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:07.209878 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:07.210132 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.210144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.211093 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.211097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.211966 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.211977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.212804 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.212811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.212990 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.213669 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:07.213688 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:07.216527 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
W20260927 22:16:07.216925 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:07.208519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:38723 (local address 127.23.173.126:37215)
0927 22:16:07.208691 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:07.208695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:07.209089 (+ 394us) server_negotiation.cc:407] Connection header received
0927 22:16:07.209245 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:07.209248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:07.209349 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:07.209432 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:07.210251 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.210980 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.212078 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.212642 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.212775 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.216706 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:38723: BlockingRecv error: recv got EOF from 127.23.173.65:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:07 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:07.678576 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:07.670040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:45065)
0927 22:16:07.670569 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:07.670584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:07.670738 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:07.671067 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:07.671071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:07.671498 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:07.671678 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.671687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.672745 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.672751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.673639 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:07.673650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.674417 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.674423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:07.674640 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.675359 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:07.675376 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:07.678301 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":317,"threads_started":1}
W20260927 22:16:07.678718 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:07.670101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:45065 (local address 127.23.173.126:37215)
0927 22:16:07.670314 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:07.670323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:07.670666 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:16:07.670857 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:07.670870 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:07.670928 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:07.671036 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:07.671833 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.672576 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.673790 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:07.674267 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:07.674402 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:07.678508 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:45065: BlockingRecv error: recv got EOF from 127.23.173.66:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:08.146345 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:08.132856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:37751)
0927 22:16:08.133338 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:08.133354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:08.133466 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:08.133751 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:08.133754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:08.134353 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:08.134578 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.134588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.135914 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.135918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.136878 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.136890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.137524 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.137528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.137643 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.143148 (+ 5505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:08.143166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:08.145963 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":267,"threads_started":1}
W20260927 22:16:08.146450 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:08.132941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:37751 (local address 127.23.173.126:37215)
0927 22:16:08.133183 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:08.133189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:08.133435 (+ 246us) server_negotiation.cc:407] Connection header received
0927 22:16:08.133565 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:08.133569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:08.133630 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:08.133712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:08.134747 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.135761 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.137001 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.137406 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.137493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.146200 (+ 8707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:37751: BlockingRecv error: recv got EOF from 127.23.173.67:37751 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:16:08.226522 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:08.217840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:47547 (local address 127.23.173.126:37215)
0927 22:16:08.218064 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:08.218073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:08.218548 (+ 475us) server_negotiation.cc:407] Connection header received
0927 22:16:08.218745 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:08.218749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:08.218879 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:08.218982 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:08.219804 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.220516 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.221873 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.222347 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.222436 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.226272 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:47547: BlockingRecv error: recv got EOF from 127.23.173.65:47547 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:16:08.226531 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:08.217813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:47547)
0927 22:16:08.218425 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:08.218459 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:08.218606 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:08.219046 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:08.219052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:08.219507 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:08.219671 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.219680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.220643 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.220647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.221758 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.221770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.222467 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.222471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.222598 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.223253 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:08.223271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:08.226092 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":366,"threads_started":1}
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:08 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:08.687830 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:08.679647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:60103)
0927 22:16:08.680154 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:08.680189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:08.680309 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:08.680589 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:08.680592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:08.680969 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:08.681118 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.681127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.682111 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.682131 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.683067 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:08.683080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.683810 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.683816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:08.684004 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.684563 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:08.684580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:08.687498 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":303,"threads_started":1}
W20260927 22:16:08.687904 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:08.679674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:60103 (local address 127.23.173.126:37215)
0927 22:16:08.679841 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:08.679846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:08.680261 (+ 415us) server_negotiation.cc:407] Connection header received
0927 22:16:08.680399 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:08.680402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:08.680462 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:08.680539 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:08.681233 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.681931 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.683213 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:08.683652 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:08.683787 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:08.687685 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:60103: BlockingRecv error: recv got EOF from 127.23.173.66:60103 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:09.155671 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:09.147328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:35265)
0927 22:16:09.147829 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:09.147844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:09.147998 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:09.148377 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:09.148382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:09.148806 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:09.148953 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.148962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.150000 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.150004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.150909 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.150919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.151843 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.151848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.152052 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.152772 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:09.152789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:09.155442 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
W20260927 22:16:09.155879 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:09.147421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:35265 (local address 127.23.173.126:37215)
0927 22:16:09.147651 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:09.147658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:09.147933 (+ 275us) server_negotiation.cc:407] Connection header received
0927 22:16:09.148122 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:09.148133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:09.148193 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:09.148312 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:09.149097 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.149827 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.151044 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.151701 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.151828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.155628 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:35265: BlockingRecv error: recv got EOF from 127.23.173.67:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:09.235572 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:09.227528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:43765)
0927 22:16:09.228064 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:09.228083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:09.228190 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:09.228474 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:09.228477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:09.228871 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:09.229024 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.229033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.229985 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.229988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.230877 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.230887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.231575 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.231579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.231690 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.232321 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:09.232338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:09.235300 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":292,"threads_started":1}
W20260927 22:16:09.235613 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:09.227580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:43765 (local address 127.23.173.126:37215)
0927 22:16:09.227736 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:09.227740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:09.228148 (+ 408us) server_negotiation.cc:407] Connection header received
0927 22:16:09.228282 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:09.228286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:09.228357 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:09.228436 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:09.229136 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.229860 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.230991 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.231432 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.231521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.235442 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:43765: BlockingRecv error: recv got EOF from 127.23.173.65:43765 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:09 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:09.697746 1063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:09.688957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:59181)
0927 22:16:09.689528 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:09.689544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:09.689694 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:09.690110 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:09.690115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:09.690584 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:09.690740 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.690749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.691773 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.691778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.692713 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:09.692725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.693605 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.693610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:09.693818 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.694548 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:09.694569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:09.697521 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
W20260927 22:16:09.697997 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:09.689143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:59181 (local address 127.23.173.126:37215)
0927 22:16:09.689375 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:09.689381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:09.689625 (+ 244us) server_negotiation.cc:407] Connection header received
0927 22:16:09.689833 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:09.689840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:09.689919 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:09.690085 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:09.690886 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.691633 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.692874 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:09.693390 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:09.693557 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:09.697709 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:59181: BlockingRecv error: recv got EOF from 127.23.173.66:59181 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:10.164727 1064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:10.156588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:52777)
0927 22:16:10.157119 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:10.157135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:10.157303 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:10.157656 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:10.157662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:10.158158 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:10.158317 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.158327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.159316 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.159325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.160259 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.160269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.161020 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.161028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.161189 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.161752 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:10.161769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:10.164496 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
W20260927 22:16:10.164914 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:10.156661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:52777 (local address 127.23.173.126:37215)
0927 22:16:10.156887 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:10.156895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:10.157226 (+ 331us) server_negotiation.cc:407] Connection header received
0927 22:16:10.157458 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:10.157462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:10.157522 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:10.157622 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:10.158460 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.159182 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.160407 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.160835 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.160961 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.164692 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:52777: BlockingRecv error: recv got EOF from 127.23.173.67:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:10.244717 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:10.236420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:43181)
0927 22:16:10.236908 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:10.236929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:10.237040 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:10.237348 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:10.237351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:10.237704 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:10.237874 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.237882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.239163 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.239168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.240401 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.240413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.241047 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.241051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.241163 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.241752 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:10.241770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:10.244485 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":268,"threads_started":1}
W20260927 22:16:10.244936 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:10.236539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:43181 (local address 127.23.173.126:37215)
0927 22:16:10.236770 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:10.236775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:10.236998 (+ 223us) server_negotiation.cc:407] Connection header received
0927 22:16:10.237135 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:10.237139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:10.237210 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:10.237297 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:10.238060 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.239026 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.240543 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.240933 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.241023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.244685 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:43181: BlockingRecv error: recv got EOF from 127.23.173.65:43181 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:10 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:10.707475 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:10.698827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:58139)
0927 22:16:10.699369 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:10.699385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:10.699536 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:10.699904 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:10.699911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:10.700286 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:10.700466 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.700475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.701529 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.701535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.702512 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:10.702525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.703262 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.703267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:10.703456 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.704129 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:10.704146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:10.707235 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":263,"threads_started":1}
W20260927 22:16:10.707647 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:10.698911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:58139 (local address 127.23.173.126:37215)
0927 22:16:10.699137 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:10.699144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:10.699473 (+ 329us) server_negotiation.cc:407] Connection header received
0927 22:16:10.699668 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:10.699673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:10.699745 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:10.699857 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:10.700616 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.701384 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.702658 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:10.703102 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:10.703237 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:10.707426 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:58139: BlockingRecv error: recv got EOF from 127.23.173.66:58139 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.67@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:11.174384 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:11.165932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.67:51217)
0927 22:16:11.166498 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:11.166523 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:11.166671 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:11.167055 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:11.167059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:11.167542 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:11.167787 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.167806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.169041 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.169046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.169945 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.170011 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.170613 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.170617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.170738 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.171298 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:11.171322 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:11.174147 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":273,"threads_started":1}
W20260927 22:16:11.174613 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:11.166018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:51217 (local address 127.23.173.126:37215)
0927 22:16:11.166270 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:11.166277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:11.166605 (+ 328us) server_negotiation.cc:407] Connection header received
0927 22:16:11.166826 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:11.166835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:11.166914 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:11.167012 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:11.167941 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.168916 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.170112 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.170497 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.170586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.174351 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.67:51217: BlockingRecv error: recv got EOF from 127.23.173.67:51217 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.65@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:11.254120 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:11.246010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.65:33371)
0927 22:16:11.246563 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:11.246583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:11.246693 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:11.246990 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:11.246993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:11.247377 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:11.247538 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.247546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.248480 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.248484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.249388 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.249398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.250247 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.250252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.250459 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.251187 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:11.251212 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:11.253846 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
W20260927 22:16:11.254299 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:11.246088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:33371 (local address 127.23.173.126:37215)
0927 22:16:11.246314 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:11.246320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:11.246662 (+ 342us) server_negotiation.cc:407] Connection header received
0927 22:16:11.246789 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:11.246793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:11.246865 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:11.246946 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:11.247655 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.248363 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.249502 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.250089 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.250224 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.254062 (+ 3838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.65:33371: BlockingRecv error: recv got EOF from 127.23.173.65:33371 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:16:11 dist-test-slave-3wc3 krb5kdc[32492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.173.66@KRBTEST.COM for oryx/127.23.173.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:16:11.716758 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:11.708606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37215 (local address 127.23.173.66:45489)
0927 22:16:11.709172 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:11.709187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:11.709294 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:11.709573 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:11.709576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:11.709990 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:11.710157 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.710166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.711110 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.711113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.712037 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:11.712048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.712941 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.712947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:11.713137 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.713799 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:11.713818 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:11.716503 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.173.126:37215: Server oryx/127.23.173.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
W20260927 22:16:11.716895 760 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:11.708672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:45489 (local address 127.23.173.126:37215)
0927 22:16:11.708852 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:11.708857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:11.709260 (+ 403us) server_negotiation.cc:407] Connection header received
0927 22:16:11.709388 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:11.709392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:11.709452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:11.709534 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:11.710275 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.710985 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.712192 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:11.712780 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:11.712919 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:11.716681 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.173.66:45489: BlockingRecv error: recv got EOF from 127.23.173.66:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:16:11.997859 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 611
I20260927 22:16:12.009584 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:11.998912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.23.173.65:41973)
0927 22:16:11.999407 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:11.999414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:11.999440 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:16:11.999497 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:11.999504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:11.999726 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:11.999872 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:12.001290 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.002341 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.003467 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.004001 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.004098 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:12.006502 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:12.006511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:12.006523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:12.006555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:12.009064 (+ 2509us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:16:12.009325 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48086: Request ticket server kudu/127.23.173.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
W20260927 22:16:12.009706 1071 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48086: Request ticket server kudu/127.23.173.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:16:12.009792 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 611: Not authorized: Client connection negotiation failed: client connection to 127.23.173.65:41973: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:16:12.010046 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 611
I20260927 22:16:12.026711 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 762
I20260927 22:16:12.038673 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:12.027794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.23.173.66:35839)
0927 22:16:12.028468 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:12.028475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:12.028504 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:16:12.028585 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:12.028591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:12.028808 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:12.028956 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:12.030231 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.031261 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.032397 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.032933 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.033027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:12.035531 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:12.035539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:12.035554 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:12.035585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:12.038160 (+ 2575us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:16:12.038402 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41102: Request ticket server kudu/127.23.173.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":354,"threads_started":1}
W20260927 22:16:12.038794 1072 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41102: Request ticket server kudu/127.23.173.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:16:12.038806 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 762: Not authorized: Client connection negotiation failed: client connection to 127.23.173.66:35839: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:16:12.038964 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 762
I20260927 22:16:12.055404 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 893
I20260927 22:16:12.068881 1073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:12.056603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.23.173.67:34033)
0927 22:16:12.057295 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:12.057305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:12.057345 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:16:12.057426 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:12.057435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:12.057732 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:12.057945 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:12.059672 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.061342 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.062814 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.063362 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.063467 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:12.065787 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:12.065795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:12.065808 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:12.065840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:12.068403 (+ 2563us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:16:12.068647 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51962: Request ticket server kudu/127.23.173.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":342,"threads_started":1}
W20260927 22:16:12.069015 1073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51962: Request ticket server kudu/127.23.173.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:16:12.069106 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 893: Not authorized: Client connection negotiation failed: client connection to 127.23.173.67:34033: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.173.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:16:12.069291 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 893
I20260927 22:16:12.086544 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 542
I20260927 22:16:12.087729 604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:12.205402 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 542
2026-09-27T22:16:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21074 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:16:12Z Disabled control of system clock
I20260927 22:16:12.284092 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40017
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:40017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:16:12.608526 1078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:12.608825 1078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:12.609005 1078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:12.616873 1078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:16:12.616988 1078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:12.617034 1078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:12.617111 1078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:16:12.617185 1078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:16:12.626168 1078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:40017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:40017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:12.627439 1078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:12.629248 1078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:12.641716 1086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:12.641943 1078 server_base.cc:1034] running on GCE node
W20260927 22:16:12.641779 1084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:12.641755 1083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:12.643014 1078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:12.644387 1078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:12.645644 1078 hybrid_clock.cc:648] HybridClock initialized: now 1790547372645598 us; error 35 us; skew 500 ppm
I20260927 22:16:12.646096 1078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:12.648381 1078 webserver.cc:460] Webserver started at http://127.23.173.126:36471/ using document root <none> and password file <none>
I20260927 22:16:12.649082 1078 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:12.649210 1078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:12.649533 1078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:12.652429 1078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c2f31327b00d4c47aaa32b51a519d0f2"
format_stamp: "Formatted at 2026-09-27 22:16:12 on dist-test-slave-3wc3"
server_key: "bb32ce30c885841c0d200d2ffe950561"
server_key_iv: "8a98e6b3de1abf2bb7748347830f6387"
server_key_version: "encryptionkey@0"
I20260927 22:16:12.653223 1078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c2f31327b00d4c47aaa32b51a519d0f2"
format_stamp: "Formatted at 2026-09-27 22:16:12 on dist-test-slave-3wc3"
server_key: "bb32ce30c885841c0d200d2ffe950561"
server_key_iv: "8a98e6b3de1abf2bb7748347830f6387"
server_key_version: "encryptionkey@0"
I20260927 22:16:12.658362 1078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:16:12.661703 1092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:12.663092 1078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:16:12.663276 1078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c2f31327b00d4c47aaa32b51a519d0f2"
format_stamp: "Formatted at 2026-09-27 22:16:12 on dist-test-slave-3wc3"
server_key: "bb32ce30c885841c0d200d2ffe950561"
server_key_iv: "8a98e6b3de1abf2bb7748347830f6387"
server_key_version: "encryptionkey@0"
I20260927 22:16:12.663501 1078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:12.701558 1078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:12.702497 1078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:12.702811 1078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:12.721498 1078 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:40017
I20260927 22:16:12.721520 1143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:40017 every 8 connection(s)
I20260927 22:16:12.722918 1078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:16:12.725452 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1078
I20260927 22:16:12.725677 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:16:12.726557 24245 external_mini_cluster.cc:1428] Setting key 9118e41ae2afae36270a2705d4bf2f4b
I20260927 22:16:12.727371 1144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:12.742571 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:12.728830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.23.173.126:40017)
0927 22:16:12.729869 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:12.729883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:12.729933 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:16:12.732317 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:12.732345 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:12.732981 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:12.733433 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:12.734028 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.736036 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.737418 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:12.738517 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:12.738709 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:12.739784 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:12.739798 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:12.739824 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:12.740563 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:12.740677 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:12.740701 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:12.740797 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:12.740973 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 22:16:12.741275 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":309,"threads_started":1}
I20260927 22:16:12.742785 1144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2: Bootstrap starting.
I20260927 22:16:12.745775 1144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:12.746769 1144 log.cc:824] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:12.749197 1144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2: No bootstrap required, opened a new log
I20260927 22:16:12.756071 1144 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } }
I20260927 22:16:12.756453 1144 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:12.756569 1144 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f31327b00d4c47aaa32b51a519d0f2, State: Initialized, Role: FOLLOWER
I20260927 22:16:12.757158 1144 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } }
I20260927 22:16:12.757408 1144 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:16:12.757553 1144 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:16:12.757718 1144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:12.759852 1144 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } }
I20260927 22:16:12.760402 1144 leader_election.cc:302] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f31327b00d4c47aaa32b51a519d0f2; no voters:
I20260927 22:16:12.760996 1144 leader_election.cc:288] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:16:12.761108 1149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:16:12.762490 1149 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [term 1 LEADER]: Becoming Leader. State: Replica: c2f31327b00d4c47aaa32b51a519d0f2, State: Running, Role: LEADER
I20260927 22:16:12.762984 1149 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } }
I20260927 22:16:12.763106 1144 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:16:12.765730 1151 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f31327b00d4c47aaa32b51a519d0f2. Latest consensus state: current_term: 1 leader_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } } }
I20260927 22:16:12.766145 1151 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:12.766718 1150 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f31327b00d4c47aaa32b51a519d0f2" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 40017 } } }
I20260927 22:16:12.767042 1150 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:12.768590 1158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:16:12.773234 1158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:16:12.781694 1158 catalog_manager.cc:1347] Generated new cluster ID: dc0e71416a1742a1bcfef2059533b6f5
I20260927 22:16:12.781867 1158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:16:12.806207 1158 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:16:12.807525 1158 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:16:12.814429 1158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f31327b00d4c47aaa32b51a519d0f2: Generated new TSK 0
I20260927 22:16:12.815233 1158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:16:12.828904 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--builtin_ntp_servers=127.23.173.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:16:13.150520 1168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:13.150794 1168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:13.150969 1168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:13.158046 1168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:13.158138 1168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:13.158243 1168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:16:13.167502 1168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:13.168897 1168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:13.170692 1168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:13.183632 1176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:13.183576 1173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:13.183669 1174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:13.184656 1168 server_base.cc:1034] running on GCE node
I20260927 22:16:13.185273 1168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:13.186192 1168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:13.187399 1168 hybrid_clock.cc:648] HybridClock initialized: now 1790547373187356 us; error 30 us; skew 500 ppm
I20260927 22:16:13.187716 1168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:13.196202 1168 webserver.cc:460] Webserver started at http://127.23.173.65:45051/ using document root <none> and password file <none>
I20260927 22:16:13.196944 1168 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:13.197072 1168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:13.197407 1168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:13.200281 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "5ff624b923924088825b0183472b7d03"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "9b25e15eb1f66971d2928051f5096f6b"
server_key_iv: "2d3b520b95ae714868308c6f5dd9ca74"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.201303 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "5ff624b923924088825b0183472b7d03"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "9b25e15eb1f66971d2928051f5096f6b"
server_key_iv: "2d3b520b95ae714868308c6f5dd9ca74"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.209156 1168 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.006s
I20260927 22:16:13.212455 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.213719 1168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:16:13.213851 1168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "5ff624b923924088825b0183472b7d03"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "9b25e15eb1f66971d2928051f5096f6b"
server_key_iv: "2d3b520b95ae714868308c6f5dd9ca74"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.214092 1168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:13.224844 1168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:13.225565 1168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:13.225808 1168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:13.226737 1168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:13.228077 1168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:13.228161 1168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.228236 1168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:13.228302 1168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.259850 1168 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:39101
I20260927 22:16:13.259897 1294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:39101 every 8 connection(s)
I20260927 22:16:13.261250 1168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:16:13.268643 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1168
I20260927 22:16:13.268822 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:16:13.269505 24245 external_mini_cluster.cc:1428] Setting key b10fcb749bdc435bf8b8aa7bdf234541
I20260927 22:16:13.272090 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--builtin_ntp_servers=127.23.173.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:16:13.274348 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:13.263838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:41771 (local address 127.23.173.126:40017)
0927 22:16:13.264254 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:13.264259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:13.265426 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 22:16:13.267509 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:13.267515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:13.267594 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:13.267687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:13.268888 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.269718 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.271578 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.271998 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.272137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:13.273497 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:13.273504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:13.273509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:13.273536 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:13.273581 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:13.273593 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:13.273688 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:13.273912 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:16:13.274027 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":221,"threads_started":1}
I20260927 22:16:13.275327 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:13.264436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40017 (local address 127.23.173.65:41771)
0927 22:16:13.265275 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:13.265310 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:13.267239 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:13.267937 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:13.267948 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:13.268099 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:13.268682 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:13.268698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.269880 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.269884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:13.271432 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:13.271444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.272224 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.272233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:13.272479 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:13.273178 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:13.273180 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:13.273225 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:13.273304 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:13.273687 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:13.273699 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:13.273708 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:16:13.273903 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 22:16:13.274305 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260927 22:16:13.276853 1295 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40017
I20260927 22:16:13.277170 1295 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:13.277884 1295 heartbeater.cc:511] Master 127.23.173.126:40017 requested a full tablet report, sending...
I20260927 22:16:13.280067 1109 ts_manager.cc:194] Registered new tserver with Master: 5ff624b923924088825b0183472b7d03 (127.23.173.65:39101)
I20260927 22:16:13.282289 1109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.65:41771
W20260927 22:16:13.589736 1299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:13.590044 1299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:13.590139 1299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:13.597180 1299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:13.597267 1299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:13.597376 1299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:16:13.606606 1299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:13.607789 1299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:13.609467 1299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:13.622244 1305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:13.622193 1307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:13.622277 1304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:13.623099 1299 server_base.cc:1034] running on GCE node
I20260927 22:16:13.623667 1299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:13.624543 1299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:13.625765 1299 hybrid_clock.cc:648] HybridClock initialized: now 1790547373625732 us; error 50 us; skew 500 ppm
I20260927 22:16:13.626137 1299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:13.628260 1299 webserver.cc:460] Webserver started at http://127.23.173.66:41033/ using document root <none> and password file <none>
I20260927 22:16:13.628919 1299 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:13.629065 1299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:13.629370 1299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:13.633651 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "b6d57b57d7e84b98b17a4c2a225d2d6a"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "6c3056e81d94dc5d85e094dc016595d9"
server_key_iv: "d18c451c5f30a6b9e0f01c8b442ee86f"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.634486 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "b6d57b57d7e84b98b17a4c2a225d2d6a"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "6c3056e81d94dc5d85e094dc016595d9"
server_key_iv: "d18c451c5f30a6b9e0f01c8b442ee86f"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.639465 1299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:16:13.642607 1313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.643750 1299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:16:13.643913 1299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "b6d57b57d7e84b98b17a4c2a225d2d6a"
format_stamp: "Formatted at 2026-09-27 22:16:13 on dist-test-slave-3wc3"
server_key: "6c3056e81d94dc5d85e094dc016595d9"
server_key_iv: "d18c451c5f30a6b9e0f01c8b442ee86f"
server_key_version: "encryptionkey@0"
I20260927 22:16:13.644111 1299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:13.656575 1299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:13.657382 1299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:13.657671 1299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:13.658608 1299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:13.659953 1299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:13.660071 1299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.660202 1299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:13.660295 1299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:13.692821 1299 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:41155
I20260927 22:16:13.692875 1425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:41155 every 8 connection(s)
I20260927 22:16:13.694159 1299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:16:13.696545 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1299
I20260927 22:16:13.696712 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:16:13.697250 24245 external_mini_cluster.cc:1428] Setting key 461a7cc237bef677afcabef62b4fbff3
I20260927 22:16:13.700242 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--builtin_ntp_servers=127.23.173.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:16:13.707090 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:13.696776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:35713 (local address 127.23.173.126:40017)
0927 22:16:13.696953 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:13.696958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:13.698754 (+ 1796us) server_negotiation.cc:407] Connection header received
0927 22:16:13.700767 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:13.700773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:13.700867 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:13.700974 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:13.702121 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.702836 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.704369 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.704758 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.704902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:13.706360 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:13.706365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:13.706368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:13.706388 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:13.706418 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:13.706429 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:13.706502 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:13.706779 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:16:13.706866 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:16:13.708526 1428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:13.697473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40017 (local address 127.23.173.66:35713)
0927 22:16:13.698582 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:13.698635 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:13.700509 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:13.701211 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:13.701219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:13.701358 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:13.701915 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:13.701931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.702995 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.703000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:13.704215 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:13.704227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:13.704984 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:13.704990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:13.705220 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:13.705989 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:13.705994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:13.706048 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:13.706153 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:13.706545 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:13.706557 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:13.706567 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:16:13.706753 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 22:16:13.707183 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":394,"threads_started":1}
I20260927 22:16:13.709928 1426 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40017
I20260927 22:16:13.710297 1426 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:13.711023 1426 heartbeater.cc:511] Master 127.23.173.126:40017 requested a full tablet report, sending...
I20260927 22:16:13.712482 1109 ts_manager.cc:194] Registered new tserver with Master: b6d57b57d7e84b98b17a4c2a225d2d6a (127.23.173.66:41155)
I20260927 22:16:13.713255 1109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.66:35713
W20260927 22:16:14.019073 1429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:14.019332 1429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:14.019456 1429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:14.026600 1429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:14.026691 1429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:14.026870 1429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:16:14.035660 1429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:14.036906 1429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:14.038762 1429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:14.051295 1434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:14.051395 1435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:14.051263 1437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:14.053184 1429 server_base.cc:1034] running on GCE node
I20260927 22:16:14.054018 1429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:14.056000 1429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:14.057226 1429 hybrid_clock.cc:648] HybridClock initialized: now 1790547374057111 us; error 100 us; skew 500 ppm
I20260927 22:16:14.057621 1429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:14.059935 1429 webserver.cc:460] Webserver started at http://127.23.173.67:39317/ using document root <none> and password file <none>
I20260927 22:16:14.060633 1429 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:14.060758 1429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:14.061096 1429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:14.064153 1429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "fce050a69d0148ecb7e7b82d8319d018"
format_stamp: "Formatted at 2026-09-27 22:16:14 on dist-test-slave-3wc3"
server_key: "90b08ce951aac2e09718b64557f2d27b"
server_key_iv: "0af5840cf36f361b1af0ef2cbff80b1d"
server_key_version: "encryptionkey@0"
I20260927 22:16:14.064992 1429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "fce050a69d0148ecb7e7b82d8319d018"
format_stamp: "Formatted at 2026-09-27 22:16:14 on dist-test-slave-3wc3"
server_key: "90b08ce951aac2e09718b64557f2d27b"
server_key_iv: "0af5840cf36f361b1af0ef2cbff80b1d"
server_key_version: "encryptionkey@0"
I20260927 22:16:14.070300 1429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:16:14.073483 1443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:14.074790 1429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:16:14.074966 1429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fce050a69d0148ecb7e7b82d8319d018"
format_stamp: "Formatted at 2026-09-27 22:16:14 on dist-test-slave-3wc3"
server_key: "90b08ce951aac2e09718b64557f2d27b"
server_key_iv: "0af5840cf36f361b1af0ef2cbff80b1d"
server_key_version: "encryptionkey@0"
I20260927 22:16:14.075176 1429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:14.105705 1429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:14.106580 1429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:14.106885 1429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:14.107815 1429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:14.109205 1429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:14.109347 1429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:14.109476 1429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:14.109565 1429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:14.142251 1429 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:42889
I20260927 22:16:14.142295 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:42889 every 8 connection(s)
I20260927 22:16:14.143510 1429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:16:14.153663 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1429
I20260927 22:16:14.153787 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:16:14.154410 24245 external_mini_cluster.cc:1428] Setting key ba9aa6c37b80e8cabd329c6f7dd8f851
I20260927 22:16:14.155509 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:14.145919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:48649 (local address 127.23.173.126:40017)
0927 22:16:14.146122 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:14.146127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:14.147555 (+ 1428us) server_negotiation.cc:407] Connection header received
0927 22:16:14.149297 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:14.149301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:14.149370 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:14.149461 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:14.150483 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.151228 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.152770 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.153175 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.153267 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:14.154636 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:14.154643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:14.154648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:14.154669 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:14.154711 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:14.154728 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:14.154836 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:14.155059 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 22:16:14.155155 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:16:14.156558 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:14.146472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:40017 (local address 127.23.173.67:48649)
0927 22:16:14.147408 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:14.147443 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:14.149057 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:14.149641 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:14.149648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:14.149726 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:14.150342 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:14.150358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.151363 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.151367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:14.152642 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:14.152653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.153367 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.153371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:14.153551 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:14.154302 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:14.154305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:14.154341 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:14.154411 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:14.154831 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:14.154839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:14.154844 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:16:14.155017 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 22:16:14.155332 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260927 22:16:14.157934 1556 heartbeater.cc:347] Connected to a master server at 127.23.173.126:40017
I20260927 22:16:14.158227 1556 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:14.158895 1556 heartbeater.cc:511] Master 127.23.173.126:40017 requested a full tablet report, sending...
I20260927 22:16:14.160104 1108 ts_manager.cc:194] Registered new tserver with Master: fce050a69d0148ecb7e7b82d8319d018 (127.23.173.67:42889)
I20260927 22:16:14.160969 1108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.67:48649
I20260927 22:16:14.169736 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:16:14.175834 1297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:14.174195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.23.173.126:40017)
0927 22:16:14.174402 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:14.174409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:14.174950 (+ 541us) server_negotiation.cc:407] Connection header received
0927 22:16:14.175065 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:14.175069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:14.175140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:14.175221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:14.175554 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60270: BlockingRecv error: recv got EOF from 127.0.0.1:60270 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:16:14.176816 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1168
I20260927 22:16:14.184441 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:14.177651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.23.173.65:39101)
0927 22:16:14.178215 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:14.178222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:14.178247 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:16:14.178307 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:14.178313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:14.178538 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:14.178675 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:14.179106 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.180241 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.181408 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.181937 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.182093 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:14.183085 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:14.183093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:14.183105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:14.183663 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:14.183704 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:14.183722 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:14.183819 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:14.183957 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:16:14.184087 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":323,"threads_started":1}
I20260927 22:16:14.185611 1290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:14.273490 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1168
I20260927 22:16:14.290359 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1299
I20260927 22:16:14.297514 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:14.291188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.23.173.66:41155)
0927 22:16:14.291705 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:14.291712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:14.291739 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:16:14.291803 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:14.291809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:14.292025 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:14.292150 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:14.292476 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.293425 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.294636 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.295186 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.295286 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:14.296249 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:14.296256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:14.296268 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:14.296836 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:14.296879 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:14.296897 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:14.296986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:14.297104 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:16:14.297208 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":263,"threads_started":1}
I20260927 22:16:14.299475 1421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:14.380383 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1299
I20260927 22:16:14.396147 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1429
I20260927 22:16:14.404353 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:14.397279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.23.173.67:42889)
0927 22:16:14.397915 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:14.397922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:14.398003 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:16:14.398083 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:14.398090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:14.398314 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:14.398456 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:14.398835 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.399891 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.401072 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:14.401637 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:14.401735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:14.402882 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:14.402891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:14.402905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:14.403520 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:14.403569 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:14.403600 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:14.403688 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:14.403810 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 22:16:14.403959 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":333,"threads_started":1}
I20260927 22:16:14.405608 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:14.490593 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1429
I20260927 22:16:14.506421 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1078
I20260927 22:16:14.507609 1139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:14.611402 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1078
2026-09-27T22:16:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2400 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:16:14Z Disabled control of system clock
I20260927 22:16:14.681743 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43851
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:16:15.002622 1575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:15.002900 1575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:15.003083 1575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:15.010228 1575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:16:15.010327 1575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:15.010411 1575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:15.010524 1575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:16:15.010608 1575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:16:15.019575 1575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:43851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:15.020884 1575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:15.022735 1575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:15.034210 1580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.034157 1581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.034350 1583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:15.035077 1575 server_base.cc:1034] running on GCE node
I20260927 22:16:15.035882 1575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:15.037161 1575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:15.038388 1575 hybrid_clock.cc:648] HybridClock initialized: now 1790547375038357 us; error 51 us; skew 500 ppm
I20260927 22:16:15.038718 1575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:15.040908 1575 webserver.cc:460] Webserver started at http://127.23.173.126:36591/ using document root <none> and password file <none>
I20260927 22:16:15.041577 1575 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:15.041711 1575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:15.042114 1575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:15.044901 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c1f553d97d1a4faf8f6818b7965b7932"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "19e80e8a59a6349158752ba5cb7bc091"
server_key_iv: "d9c47da4e36f2e4f7c2f91369e66a2b1"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.045684 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c1f553d97d1a4faf8f6818b7965b7932"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "19e80e8a59a6349158752ba5cb7bc091"
server_key_iv: "d9c47da4e36f2e4f7c2f91369e66a2b1"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.050631 1575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:16:15.054141 1589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:15.055399 1575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:16:15.055578 1575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c1f553d97d1a4faf8f6818b7965b7932"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "19e80e8a59a6349158752ba5cb7bc091"
server_key_iv: "d9c47da4e36f2e4f7c2f91369e66a2b1"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.055788 1575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:15.061208 1575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:15.061992 1575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:15.062304 1575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:15.080258 1575 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:43851
I20260927 22:16:15.080307 1640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:43851 every 8 connection(s)
I20260927 22:16:15.081699 1575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:16:15.085863 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:15.091428 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1575
I20260927 22:16:15.091643 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:16:15.092471 24245 external_mini_cluster.cc:1428] Setting key 33c224a0738c1ebb725f018fe151eabb
I20260927 22:16:15.096854 1641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932: Bootstrap starting.
I20260927 22:16:15.100309 1641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:15.100593 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:15.094539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.23.173.126:43851)
0927 22:16:15.095551 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:15.095562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:15.095602 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:16:15.097212 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:15.097232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:15.097637 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:15.097937 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:15.099019 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:15.099025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:15.099037 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:15.099050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:15.099107 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:15.099125 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:15.099297 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:15.099427 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:16:15.099696 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":398,"threads_started":1}
I20260927 22:16:15.101377 1641 log.cc:824] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:15.104023 1641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932: No bootstrap required, opened a new log
I20260927 22:16:15.110836 1641 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } }
I20260927 22:16:15.111191 1641 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:15.111292 1641 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f553d97d1a4faf8f6818b7965b7932, State: Initialized, Role: FOLLOWER
I20260927 22:16:15.111900 1641 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } }
I20260927 22:16:15.112145 1641 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:16:15.112298 1641 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:16:15.112484 1641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:15.114658 1641 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } }
I20260927 22:16:15.115127 1641 leader_election.cc:302] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f553d97d1a4faf8f6818b7965b7932; no voters:
I20260927 22:16:15.115669 1641 leader_election.cc:288] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:16:15.115803 1646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:16:15.116532 1646 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [term 1 LEADER]: Becoming Leader. State: Replica: c1f553d97d1a4faf8f6818b7965b7932, State: Running, Role: LEADER
I20260927 22:16:15.117110 1646 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } }
I20260927 22:16:15.117945 1641 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:16:15.119918 1647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f553d97d1a4faf8f6818b7965b7932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } } }
I20260927 22:16:15.119937 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f553d97d1a4faf8f6818b7965b7932. Latest consensus state: current_term: 1 leader_uuid: "c1f553d97d1a4faf8f6818b7965b7932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f553d97d1a4faf8f6818b7965b7932" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43851 } } }
I20260927 22:16:15.120326 1647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:15.120424 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:15.122380 1651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:16:15.127784 1651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:16:15.136526 1651 catalog_manager.cc:1347] Generated new cluster ID: 989783a4dcd545f5a5cb109681e08b4d
I20260927 22:16:15.136627 1651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:16:15.158066 1651 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:16:15.158506 1651 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:16:15.165592 1651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1f553d97d1a4faf8f6818b7965b7932: Generated new TSK 0
I20260927 22:16:15.166625 1651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:16:15.182323 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--builtin_ntp_servers=127.23.173.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:16:15.514453 1665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:15.514812 1665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:15.514984 1665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:15.522562 1665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:15.522682 1665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:15.522854 1665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:16:15.531836 1665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:15.533002 1665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:15.534787 1665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:15.547780 1670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.547870 1671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.547873 1673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:15.551241 1665 server_base.cc:1034] running on GCE node
I20260927 22:16:15.551856 1665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:15.552743 1665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:15.553943 1665 hybrid_clock.cc:648] HybridClock initialized: now 1790547375553882 us; error 54 us; skew 500 ppm
I20260927 22:16:15.554314 1665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:15.556406 1665 webserver.cc:460] Webserver started at http://127.23.173.65:33893/ using document root <none> and password file <none>
I20260927 22:16:15.557070 1665 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:15.557180 1665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:15.557502 1665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:15.560530 1665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "9fed87bd24a64dda9498d69e8bbcb68d"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "166ff579e098dbdda54477d45f4b5d5f"
server_key_iv: "8de0c8da9357eae2b7c4980325ec64b3"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.561308 1665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "9fed87bd24a64dda9498d69e8bbcb68d"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "166ff579e098dbdda54477d45f4b5d5f"
server_key_iv: "8de0c8da9357eae2b7c4980325ec64b3"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.566578 1665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:16:15.569751 1679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:15.571038 1665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:16:15.571205 1665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "9fed87bd24a64dda9498d69e8bbcb68d"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "166ff579e098dbdda54477d45f4b5d5f"
server_key_iv: "8de0c8da9357eae2b7c4980325ec64b3"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.571426 1665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:15.576833 1665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:15.577567 1665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:15.577823 1665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:15.578773 1665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:15.580083 1665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:15.580202 1665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:15.580348 1665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:15.580439 1665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:15.612977 1665 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:39013
I20260927 22:16:15.613029 1791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:39013 every 8 connection(s)
I20260927 22:16:15.614411 1665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:16:15.617175 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1665
I20260927 22:16:15.617372 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:16:15.618264 24245 external_mini_cluster.cc:1428] Setting key 3c45df53cab2f1f78f6e5dfe75617775
I20260927 22:16:15.621025 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--builtin_ntp_servers=127.23.173.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:16:15.623481 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:15.616901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:49099 (local address 127.23.173.126:43851)
0927 22:16:15.617378 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:15.617383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:15.619020 (+ 1637us) server_negotiation.cc:407] Connection header received
0927 22:16:15.620878 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:15.620883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:15.620945 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:15.621082 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:15.622760 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:15.622764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:15.622767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:15.622782 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:15.622825 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:15.622838 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:15.622935 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:15.623172 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:16:15.623221 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":250,"threads_started":1}
I20260927 22:16:15.624655 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:15.617621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43851 (local address 127.23.173.65:49099)
0927 22:16:15.618788 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:15.618883 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:15.620621 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:15.621480 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:15.621489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:15.621577 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:15.622402 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:15.622407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:15.622459 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:15.622589 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:15.622938 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:15.622969 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:15.622981 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:16:15.623261 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 22:16:15.623503 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":355,"threads_started":1}
I20260927 22:16:15.626127 1792 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43851
I20260927 22:16:15.626418 1792 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:15.626986 1792 heartbeater.cc:511] Master 127.23.173.126:43851 requested a full tablet report, sending...
I20260927 22:16:15.629205 1606 ts_manager.cc:194] Registered new tserver with Master: 9fed87bd24a64dda9498d69e8bbcb68d (127.23.173.65:39013)
W20260927 22:16:15.945436 1796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:15.945698 1796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:15.945784 1796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:15.953047 1796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:15.953150 1796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:15.953325 1796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:16:15.962584 1796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:15.963845 1796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:15.965600 1796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:15.978670 1802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.978705 1801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:15.978670 1804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:15.979523 1796 server_base.cc:1034] running on GCE node
I20260927 22:16:15.980041 1796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:15.980873 1796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:15.982059 1796 hybrid_clock.cc:648] HybridClock initialized: now 1790547375982015 us; error 32 us; skew 500 ppm
I20260927 22:16:15.982339 1796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:15.984481 1796 webserver.cc:460] Webserver started at http://127.23.173.66:39045/ using document root <none> and password file <none>
I20260927 22:16:15.985106 1796 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:15.985188 1796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:15.985443 1796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:15.988161 1796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "e401af6390ec45b3afbf6fb045cbe36b"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "8915e36bdf94372e1cfeb10382c8f206"
server_key_iv: "526cec035fe72aa80ca6cedd109000d5"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.988885 1796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "e401af6390ec45b3afbf6fb045cbe36b"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "8915e36bdf94372e1cfeb10382c8f206"
server_key_iv: "526cec035fe72aa80ca6cedd109000d5"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.993687 1796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:16:15.997176 1810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:15.998555 1796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:16:15.998728 1796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "e401af6390ec45b3afbf6fb045cbe36b"
format_stamp: "Formatted at 2026-09-27 22:16:15 on dist-test-slave-3wc3"
server_key: "8915e36bdf94372e1cfeb10382c8f206"
server_key_iv: "526cec035fe72aa80ca6cedd109000d5"
server_key_version: "encryptionkey@0"
I20260927 22:16:15.998960 1796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:16.004271 1796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:16.005070 1796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:16.005336 1796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:16.006279 1796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:16.007589 1796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:16.007723 1796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:16.007850 1796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:16.007936 1796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:16.038990 1796 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:35731
I20260927 22:16:16.039041 1922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:35731 every 8 connection(s)
I20260927 22:16:16.040268 1796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:16:16.045421 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1796
I20260927 22:16:16.045610 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:16:16.046099 24245 external_mini_cluster.cc:1428] Setting key a33fc941f5be1d0436d49b29a8e2d82c
I20260927 22:16:16.048516 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.042889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:42299 (local address 127.23.173.126:43851)
0927 22:16:16.043063 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.043068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.044440 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 22:16:16.046231 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.046237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.046309 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.046426 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.047740 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:16.047744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:16.047748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:16.047762 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:16.047795 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:16.047806 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:16.047972 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:16.048177 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:16:16.048244 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:16:16.049176 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--builtin_ntp_servers=127.23.173.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:16:16.049492 1925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.043427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43851 (local address 127.23.173.66:42299)
0927 22:16:16.044294 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:16.044328 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:16.045899 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:16.046628 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:16.046636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:16.046724 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:16.047421 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:16.047424 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:16.047469 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:16.047575 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:16.047938 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:16.047948 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:16.047956 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:16:16.048262 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:16:16.048492 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":256,"threads_started":1}
I20260927 22:16:16.050806 1923 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43851
I20260927 22:16:16.051120 1923 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:16.051759 1923 heartbeater.cc:511] Master 127.23.173.126:43851 requested a full tablet report, sending...
I20260927 22:16:16.053598 1606 ts_manager.cc:194] Registered new tserver with Master: e401af6390ec45b3afbf6fb045cbe36b (127.23.173.66:35731)
W20260927 22:16:16.377159 1926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:16.377463 1926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:16.377620 1926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:16.384785 1926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:16.384930 1926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:16.385118 1926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:16:16.394279 1926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:16.395627 1926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:16.397403 1926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:16.410681 1932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:16.410645 1934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:16.410724 1931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:16.411295 1926 server_base.cc:1034] running on GCE node
I20260927 22:16:16.411942 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:16.412921 1926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:16.414151 1926 hybrid_clock.cc:648] HybridClock initialized: now 1790547376414086 us; error 52 us; skew 500 ppm
I20260927 22:16:16.414481 1926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:16:16.416788 1926 webserver.cc:460] Webserver started at http://127.23.173.67:35415/ using document root <none> and password file <none>
I20260927 22:16:16.417493 1926 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:16.417635 1926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:16.418035 1926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:16.420989 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "af2f3a2112004dae8cef947ee98b3497"
format_stamp: "Formatted at 2026-09-27 22:16:16 on dist-test-slave-3wc3"
server_key: "bae8ed53daa2408d0668606467b49abb"
server_key_iv: "51a869b4fecba8dded443a8fe40b7e2f"
server_key_version: "encryptionkey@0"
I20260927 22:16:16.421762 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "af2f3a2112004dae8cef947ee98b3497"
format_stamp: "Formatted at 2026-09-27 22:16:16 on dist-test-slave-3wc3"
server_key: "bae8ed53daa2408d0668606467b49abb"
server_key_iv: "51a869b4fecba8dded443a8fe40b7e2f"
server_key_version: "encryptionkey@0"
I20260927 22:16:16.427062 1926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:16:16.430495 1940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:16.431902 1926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:16:16.432072 1926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "af2f3a2112004dae8cef947ee98b3497"
format_stamp: "Formatted at 2026-09-27 22:16:16 on dist-test-slave-3wc3"
server_key: "bae8ed53daa2408d0668606467b49abb"
server_key_iv: "51a869b4fecba8dded443a8fe40b7e2f"
server_key_version: "encryptionkey@0"
I20260927 22:16:16.432300 1926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:16.438127 1926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:16.438915 1926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:16.439183 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:16.440070 1926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:16.441440 1926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:16.441550 1926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:16.441675 1926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:16.441764 1926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:16.474515 1926 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:39047
I20260927 22:16:16.474551 2052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:39047 every 8 connection(s)
I20260927 22:16:16.475952 1926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:16:16.484004 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.478504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:37839 (local address 127.23.173.126:43851)
0927 22:16:16.478676 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.478681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.480071 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 22:16:16.481748 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.481753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.481824 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.481943 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.483251 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:16.483256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:16.483260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:16.483278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:16.483316 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:16.483329 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:16.483450 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:16.483635 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 22:16:16.483707 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:16:16.484473 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 1926
I20260927 22:16:16.484660 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:16:16.484994 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.479050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43851 (local address 127.23.173.67:37839)
0927 22:16:16.479925 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:16.479960 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:16.481480 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:16.482199 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:16.482207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:16.482294 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:16.482964 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:16:16.482967 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:16.483005 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:16:16.483085 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:16.483422 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:16.483432 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:16.483438 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:16:16.483717 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 22:16:16.483951 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260927 22:16:16.485114 24245 external_mini_cluster.cc:1428] Setting key 90c2c779f0886aa72c424a4e4d9eb091
I20260927 22:16:16.486485 2053 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43851
I20260927 22:16:16.486763 2053 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:16.487339 2053 heartbeater.cc:511] Master 127.23.173.126:43851 requested a full tablet report, sending...
I20260927 22:16:16.488689 1606 ts_manager.cc:194] Registered new tserver with Master: af2f3a2112004dae8cef947ee98b3497 (127.23.173.67:39047)
I20260927 22:16:16.500003 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:16:16.506175 1794 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:16:16.504552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.23.173.126:43851)
0927 22:16:16.504710 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.504715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.505217 (+ 502us) server_negotiation.cc:407] Connection header received
0927 22:16:16.505415 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.505427 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.505478 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.505582 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.505927 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55136: BlockingRecv error: recv got EOF from 127.0.0.1:55136 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:16:16.507112 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1665
I20260927 22:16:16.510416 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.508029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.23.173.65:39013)
0927 22:16:16.508525 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.508534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.508570 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:16:16.508636 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.508641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.508849 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.508968 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.509777 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:16.509783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:16.509795 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:16.509812 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:16.509852 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:16.509866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:16.510012 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:16.510107 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:16:16.510184 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":261,"threads_started":1}
I20260927 22:16:16.511623 1787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:16.592542 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1665
I20260927 22:16:16.609109 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1796
I20260927 22:16:16.612828 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.610145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.23.173.66:35731)
0927 22:16:16.610653 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.610660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.610687 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:16:16.610756 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.610762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.610986 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.611115 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.612010 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:16.612016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:16.612028 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:16.612044 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:16.612084 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:16.612098 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:16.612202 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:16.612420 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 22:16:16.612547 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260927 22:16:16.614081 1918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:16.694294 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1796
I20260927 22:16:16.710160 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1926
I20260927 22:16:16.714102 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:16.711113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.23.173.67:39047)
0927 22:16:16.711686 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:16.711693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:16.711722 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:16:16.711784 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:16.711790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:16.712026 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:16.712199 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:16.713307 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:16.713313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:16.713324 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:16:16.713335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:16.713376 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:16:16.713393 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:16.713573 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:16.713638 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:16:16.713743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":256,"threads_started":1}
I20260927 22:16:16.715322 2048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:16.796417 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1926
I20260927 22:16:16.812181 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 1575
I20260927 22:16:16.813372 1636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:16.915791 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 1575
2026-09-27T22:16:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2303 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:16:17 dist-test-slave-3wc3 krb5kdc[2071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:16:17 dist-test-slave-3wc3 krb5kdc[2071](info): set up 2 sockets
Sep 27 22:16:17 dist-test-slave-3wc3 krb5kdc[2071](info): commencing operation
krb5kdc: starting...
W20260927 22:16:18.062408 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:16:18 dist-test-slave-3wc3 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:16:18.289537 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34245
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:16:18.619144 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:18.619522 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:18.619701 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:18.630213 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:16:18.630347 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:18.630429 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:18.630508 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:16:18.630571 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:16:18.646384 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:34245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:18.647899 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:18.649235 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:18.661259 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:18.661288 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:18.661655 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:18.662258 2084 server_base.cc:1034] running on GCE node
I20260927 22:16:18.663056 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:18.664386 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:18.665582 2084 hybrid_clock.cc:648] HybridClock initialized: now 1790547378665536 us; error 31 us; skew 500 ppm
I20260927 22:16:18.665923 2084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:16:18 dist-test-slave-3wc3 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:16:18.668828 2084 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:16:18.670289 2084 webserver.cc:460] Webserver started at http://127.23.173.126:33251/ using document root <none> and password file <none>
I20260927 22:16:18.670970 2084 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:18.671109 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:18.671406 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:18.674423 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "29f656c2f3754a619b24ecb09e943309"
format_stamp: "Formatted at 2026-09-27 22:16:18 on dist-test-slave-3wc3"
server_key: "f0b877c188f7b8412da76275b21d420a"
server_key_iv: "a46d6840a70d96594fdda994b47894b0"
server_key_version: "encryptionkey@0"
I20260927 22:16:18.675148 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "29f656c2f3754a619b24ecb09e943309"
format_stamp: "Formatted at 2026-09-27 22:16:18 on dist-test-slave-3wc3"
server_key: "f0b877c188f7b8412da76275b21d420a"
server_key_iv: "a46d6840a70d96594fdda994b47894b0"
server_key_version: "encryptionkey@0"
I20260927 22:16:18.680099 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:16:18.683331 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:18.684525 2084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:16:18.684698 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "29f656c2f3754a619b24ecb09e943309"
format_stamp: "Formatted at 2026-09-27 22:16:18 on dist-test-slave-3wc3"
server_key: "f0b877c188f7b8412da76275b21d420a"
server_key_iv: "a46d6840a70d96594fdda994b47894b0"
server_key_version: "encryptionkey@0"
I20260927 22:16:18.684900 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:18.706862 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:18.710609 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:18.710891 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:18.731928 2084 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:34245
I20260927 22:16:18.732015 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:34245 every 8 connection(s)
I20260927 22:16:18.733585 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:16:18.741639 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 2084
I20260927 22:16:18.741884 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:16:18.742331 2151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:18.742630 24245 external_mini_cluster.cc:1428] Setting key da925deba2dd926b078d485f98376820
Sep 27 22:16:18 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:16:18.760318 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: Bootstrap starting.
I20260927 22:16:18.764757 2151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:18.764786 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:18.744572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.23.173.126:34245)
0927 22:16:18.745308 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:18.745327 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:18.745365 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:16:18.748175 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:18.748196 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:18.748609 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:18.748984 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:18.750541 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:18.751873 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:18.753106 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:18.753755 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:18.753935 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:18.757844 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:18.757852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:18.757865 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:18.757900 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:18.760936 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:18.761571 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:18.761577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:18.761582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:18.761745 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:18.762185 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:18.762199 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:18.762201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:18.762690 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:18.763028 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:18.763387 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 22:16:18.763814 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":317,"threads_started":1}
I20260927 22:16:18.766070 2151 log.cc:824] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:18.768919 2151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: No bootstrap required, opened a new log
I20260927 22:16:18.776376 2151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } }
I20260927 22:16:18.776739 2151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:18.776841 2151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f656c2f3754a619b24ecb09e943309, State: Initialized, Role: FOLLOWER
I20260927 22:16:18.777419 2151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } }
I20260927 22:16:18.777644 2151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:16:18.777784 2151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:16:18.777984 2151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:18.780120 2151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } }
I20260927 22:16:18.780586 2151 leader_election.cc:302] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f656c2f3754a619b24ecb09e943309; no voters:
I20260927 22:16:18.781186 2151 leader_election.cc:288] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:16:18.781350 2156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:16:18.782146 2156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [term 1 LEADER]: Becoming Leader. State: Replica: 29f656c2f3754a619b24ecb09e943309, State: Running, Role: LEADER
I20260927 22:16:18.782790 2156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } }
I20260927 22:16:18.783281 2151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:16:18.785745 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f656c2f3754a619b24ecb09e943309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } } }
I20260927 22:16:18.786134 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:18.791473 2158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f656c2f3754a619b24ecb09e943309. Latest consensus state: current_term: 1 leader_uuid: "29f656c2f3754a619b24ecb09e943309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f656c2f3754a619b24ecb09e943309" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34245 } } }
I20260927 22:16:18.792330 2158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309 [sys.catalog]: This master's current role is: LEADER
I20260927 22:16:18.800690 2163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:16:18.806058 2163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:16:18.814925 2163 catalog_manager.cc:1347] Generated new cluster ID: 940e00cbd3fc4002ba3f0b9ae3624090
I20260927 22:16:18.815068 2163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:16:18.831100 2163 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:16:18.832330 2163 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:16:18.844283 2163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: Generated new TSK 0
I20260927 22:16:18.845139 2163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:16:18.906066 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:16:19.229614 2177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:19.229867 2177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:19.229974 2177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:19.236884 2177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:19.236980 2177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:19.237087 2177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:16:19.246505 2177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:19.247671 2177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:19.248865 2177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:19.261821 2185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:19.261890 2183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:19.262049 2182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:19.262820 2177 server_base.cc:1034] running on GCE node
I20260927 22:16:19.263522 2177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:19.264446 2177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:19.265676 2177 hybrid_clock.cc:648] HybridClock initialized: now 1790547379265614 us; error 54 us; skew 500 ppm
I20260927 22:16:19.266060 2177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:16:19 dist-test-slave-3wc3 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547379, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:16:19.269181 2177 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:16:19.270633 2177 webserver.cc:460] Webserver started at http://127.23.173.65:34405/ using document root <none> and password file <none>
I20260927 22:16:19.271334 2177 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:19.271483 2177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:19.271778 2177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:19.274772 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "8df96cb47a39458d91638e11cb30739c"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "a1a14c8c13e072f11c309a9eb1c4e636"
server_key_iv: "558e9bd73911f330a2f3a62a96db6d9c"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.275522 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "8df96cb47a39458d91638e11cb30739c"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "a1a14c8c13e072f11c309a9eb1c4e636"
server_key_iv: "558e9bd73911f330a2f3a62a96db6d9c"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.280423 2177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:16:19.283783 2192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.285116 2177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:16:19.285292 2177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "8df96cb47a39458d91638e11cb30739c"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "a1a14c8c13e072f11c309a9eb1c4e636"
server_key_iv: "558e9bd73911f330a2f3a62a96db6d9c"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.285480 2177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:19.299573 2177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:19.303875 2177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:19.304154 2177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:19.305055 2177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:19.306402 2177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:19.306499 2177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.306645 2177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:19.306731 2177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.339295 2177 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:37853
I20260927 22:16:19.339334 2304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:37853 every 8 connection(s)
I20260927 22:16:19.340567 2177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:16:19.347712 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 2177
I20260927 22:16:19.347939 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:16:19.348737 24245 external_mini_cluster.cc:1428] Setting key 8b8b66a639ca58db361ab0b49beecc1c
Sep 27 22:16:19 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547379, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:16:19.362450 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:19.343151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:45621 (local address 127.23.173.126:34245)
0927 22:16:19.343709 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:19.343715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:19.344561 (+ 846us) server_negotiation.cc:407] Connection header received
0927 22:16:19.346448 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:19.346453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:19.346540 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:19.346656 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:19.348559 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.349301 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.351035 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.351744 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.351895 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:19.355669 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:19.355678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:19.355685 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:19.355730 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:19.359003 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:19.359837 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:19.359858 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:19.359864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:19.360010 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:19.360412 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:19.360419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:19.360425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:19.360789 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:19.360952 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:19.361672 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 22:16:19.361928 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":353,"threads_started":1}
I20260927 22:16:19.363006 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:19.343690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34245 (local address 127.23.173.65:45621)
0927 22:16:19.344399 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:19.344442 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:19.346172 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:19.346885 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:19.346893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:19.347541 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:19.348378 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:19.348405 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.349468 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.349477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:19.350889 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:19.350901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.351959 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.351963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:19.352145 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:19.352816 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:19.352837 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:19.355436 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:19.359139 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:19.359151 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:19.359172 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:19.359711 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:19.360112 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:19.360115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:19.360117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:19.360301 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:19.360936 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:19.360942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:19.361266 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:16:19.361508 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 22:16:19.361872 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":320,"threads_started":1}
I20260927 22:16:19.364482 2305 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34245
I20260927 22:16:19.364801 2305 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:19.365664 2305 heartbeater.cc:511] Master 127.23.173.126:34245 requested a full tablet report, sending...
I20260927 22:16:19.367820 2116 ts_manager.cc:194] Registered new tserver with Master: 8df96cb47a39458d91638e11cb30739c (127.23.173.65:37853)
I20260927 22:16:19.369328 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:45621
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:16:19.403661 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:16:19.718524 2311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:19.718852 2311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:19.719003 2311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:19.726241 2311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:19.726389 2311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:19.726574 2311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:16:19.736124 2311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:19.737334 2311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:19.738579 2311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:19.751240 2316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:19.751302 2319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:19.751340 2317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:19.752370 2311 server_base.cc:1034] running on GCE node
I20260927 22:16:19.752971 2311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:19.753845 2311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:19.755033 2311 hybrid_clock.cc:648] HybridClock initialized: now 1790547379754993 us; error 48 us; skew 500 ppm
I20260927 22:16:19.755327 2311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:16:19 dist-test-slave-3wc3 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547379, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:16:19.758342 2311 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:16:19.759624 2311 webserver.cc:460] Webserver started at http://127.23.173.66:38285/ using document root <none> and password file <none>
I20260927 22:16:19.760303 2311 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:19.760438 2311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:19.760716 2311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:19.763556 2311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "ea07b36d625a4cae928fa514c0955727"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "d40cd16f6dad1fe85a0d2f87e6122f22"
server_key_iv: "b0ec07db91280304237e746aa392d688"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.764300 2311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "ea07b36d625a4cae928fa514c0955727"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "d40cd16f6dad1fe85a0d2f87e6122f22"
server_key_iv: "b0ec07db91280304237e746aa392d688"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.769234 2311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:16:19.772691 2326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.773906 2311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:16:19.774093 2311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "ea07b36d625a4cae928fa514c0955727"
format_stamp: "Formatted at 2026-09-27 22:16:19 on dist-test-slave-3wc3"
server_key: "d40cd16f6dad1fe85a0d2f87e6122f22"
server_key_iv: "b0ec07db91280304237e746aa392d688"
server_key_version: "encryptionkey@0"
I20260927 22:16:19.774288 2311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:19.798224 2311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:19.801921 2311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:19.802316 2311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:19.803254 2311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:19.804741 2311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:19.804844 2311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.804975 2311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:19.805040 2311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:19.837368 2311 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:43123
I20260927 22:16:19.837405 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:43123 every 8 connection(s)
I20260927 22:16:19.838701 2311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:16:19.844998 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 2311
I20260927 22:16:19.845194 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:16:19.845615 24245 external_mini_cluster.cc:1428] Setting key fe26fb45478735c2702705adcc380508
Sep 27 22:16:19 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547379, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:16:19.860072 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:19.841131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:52743 (local address 127.23.173.126:34245)
0927 22:16:19.841302 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:19.841307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:19.842496 (+ 1189us) server_negotiation.cc:407] Connection header received
0927 22:16:19.844220 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:19.844224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:19.844291 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:19.844407 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:19.846186 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.847002 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.848696 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.849481 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.849620 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:19.853161 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:19.853170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:19.853178 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:19.853226 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:19.856846 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:19.857749 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:19.857753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:19.857755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:19.857894 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:19.858383 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:19.858386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:19.858388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:19.858639 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:19.858749 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:19.859670 (+ 921us) server_negotiation.cc:299] Negotiation successful
0927 22:16:19.859837 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:16:19.861580 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:19.841579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34245 (local address 127.23.173.66:52743)
0927 22:16:19.842343 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:19.842380 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:19.843985 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:19.844701 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:19.844715 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:19.845356 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:19.846025 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:19.846045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.847154 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.847159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:19.848551 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:19.848564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:19.849680 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:19.849685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:19.849878 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:19.850594 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:19.850615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:19.852941 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:19.856971 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:19.856983 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:19.857003 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:19.857623 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:19.858015 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:19.858023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:19.858029 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:19.858268 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:19.858770 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:19.858780 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:19.859298 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 22:16:19.859585 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 22:16:19.860147 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":321,"threads_started":1}
I20260927 22:16:19.863132 2439 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34245
I20260927 22:16:19.863497 2439 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:19.864248 2439 heartbeater.cc:511] Master 127.23.173.126:34245 requested a full tablet report, sending...
I20260927 22:16:19.865713 2116 ts_manager.cc:194] Registered new tserver with Master: ea07b36d625a4cae928fa514c0955727 (127.23.173.66:43123)
I20260927 22:16:19.867030 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:52743
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:16:19.899464 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:16:20.218343 2444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:16:20.218612 2444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:16:20.218705 2444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:16:20.225589 2444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:16:20.225682 2444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:16:20.225787 2444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:16:20.234840 2444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:34245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:16:20.235944 2444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:16:20.237113 2444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:16:20.249346 2452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:20.249428 2450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:16:20.249462 2449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:16:20.250365 2444 server_base.cc:1034] running on GCE node
I20260927 22:16:20.250893 2444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:16:20.251741 2444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:16:20.252925 2444 hybrid_clock.cc:648] HybridClock initialized: now 1790547380252869 us; error 47 us; skew 500 ppm
I20260927 22:16:20.253165 2444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:16:20 dist-test-slave-3wc3 krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547380, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:16:20.256026 2444 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:16:20.257442 2444 webserver.cc:460] Webserver started at http://127.23.173.67:36377/ using document root <none> and password file <none>
I20260927 22:16:20.258244 2444 fs_manager.cc:359] Metadata directory not provided
I20260927 22:16:20.258343 2444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:16:20.258587 2444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:16:20.261267 2444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "2173d95d5d46466398539d46dc84fe11"
format_stamp: "Formatted at 2026-09-27 22:16:20 on dist-test-slave-3wc3"
server_key: "c15feb4cc3b6014b9c3e36ab75c56be3"
server_key_iv: "d1c1f3d9c9a3501c9e9904a5d7234514"
server_key_version: "encryptionkey@0"
I20260927 22:16:20.262023 2444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "2173d95d5d46466398539d46dc84fe11"
format_stamp: "Formatted at 2026-09-27 22:16:20 on dist-test-slave-3wc3"
server_key: "c15feb4cc3b6014b9c3e36ab75c56be3"
server_key_iv: "d1c1f3d9c9a3501c9e9904a5d7234514"
server_key_version: "encryptionkey@0"
I20260927 22:16:20.266776 2444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:16:20.270123 2459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:20.271333 2444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:16:20.271507 2444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "2173d95d5d46466398539d46dc84fe11"
format_stamp: "Formatted at 2026-09-27 22:16:20 on dist-test-slave-3wc3"
server_key: "c15feb4cc3b6014b9c3e36ab75c56be3"
server_key_iv: "d1c1f3d9c9a3501c9e9904a5d7234514"
server_key_version: "encryptionkey@0"
I20260927 22:16:20.271699 2444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:16:20.301017 2444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:16:20.304916 2444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:16:20.305209 2444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:16:20.306161 2444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:16:20.307435 2444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:16:20.307538 2444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:20.307682 2444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:16:20.307760 2444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:16:20.344462 2444 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:35073
I20260927 22:16:20.344504 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:35073 every 8 connection(s)
I20260927 22:16:20.345685 2444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:16:20.351341 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 2444
I20260927 22:16:20.351549 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:16:20.351971 24245 external_mini_cluster.cc:1428] Setting key eb75c166e99c2b61b6141c815fef41c9
Sep 27 22:16:20 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547380, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:16:20.364409 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.348147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:41219 (local address 127.23.173.126:34245)
0927 22:16:20.348322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.348326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.349396 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 22:16:20.351304 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.351309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.351383 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.351500 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:20.353019 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.353801 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.355335 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.355721 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.355846 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.358978 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:20.358983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:20.358986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:20.359017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:20.361452 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.362275 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.362279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.362281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.362413 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.362818 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.362822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.362824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.363070 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:20.363227 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.363850 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.364062 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:16:20.365332 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.348619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:34245 (local address 127.23.173.67:41219)
0927 22:16:20.349251 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.349285 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.350994 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.351746 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.351754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.352318 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:16:20.352845 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.352871 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.353986 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.353992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.355193 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.355204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.355949 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.355956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.356166 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.356725 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:16:20.356745 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:16:20.358779 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:16:20.361622 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:20.361630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:20.361646 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:20.362143 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:20.362519 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:16:20.362525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:16:20.362528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:16:20.362697 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:16:20.363201 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:16:20.363208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:16:20.363508 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.363745 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.364230 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":255,"threads_started":1}
I20260927 22:16:20.366753 2572 heartbeater.cc:347] Connected to a master server at 127.23.173.126:34245
I20260927 22:16:20.366997 2572 heartbeater.cc:464] Registering TS with master...
I20260927 22:16:20.367627 2572 heartbeater.cc:511] Master 127.23.173.126:34245 requested a full tablet report, sending...
I20260927 22:16:20.368840 2116 ts_manager.cc:194] Registered new tserver with Master: 2173d95d5d46466398539d46dc84fe11 (127.23.173.67:35073)
I20260927 22:16:20.369619 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:41219
I20260927 22:16:20.372884 2305 heartbeater.cc:503] Master 127.23.173.126:34245 was elected leader, sending a full tablet report...
I20260927 22:16:20.378677 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:16:20.395256 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.383033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.23.173.126:34245)
0927 22:16:20.383196 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.383200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.383653 (+ 453us) server_negotiation.cc:407] Connection header received
0927 22:16:20.383859 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.383865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.383938 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.384034 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:20.385409 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.386196 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.387402 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.387868 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.387997 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.390307 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:20.390337 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:20.390340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:20.390374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:20.392668 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.393288 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.393295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.393298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.393435 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.393805 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.393811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.393814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.394169 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:20.394429 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.394740 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.394945 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:16:20.405485 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:16:20.412209 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:16:20.438783 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.425753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:35073 (local address 127.0.0.1:46580)
0927 22:16:20.426898 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.426942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.427145 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.429147 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.429157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.429195 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:16:20.429670 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.429685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.432309 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.432316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.435174 (+ 2858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.435193 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.437575 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.437584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.437858 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.437936 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.438100 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.438324 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":1142,"threads_started":1}
I20260927 22:16:20.440682 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.426047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.23.173.67:35073)
0927 22:16:20.428456 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.428467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.428504 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:16:20.428615 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.428625 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.428886 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.429880 (+ 994us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:16:20.430214 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.432135 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.435442 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.437297 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.439611 (+ 2314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.439917 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.440002 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.440211 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2207,"thread_start_us":2055,"threads_started":1}
I20260927 22:16:20.445451 2507 tablet_service.cc:1467] Processing CreateTablet for tablet c1a540bbd066488daab003affd34e088 (DEFAULT_TABLE table=test-table [id=cfc793943d874622a8fa54e6f7f11aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:16:20.447213 2507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a540bbd066488daab003affd34e088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:20.449783 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.427868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:37853 (local address 127.0.0.1:44982)
0927 22:16:20.439039 (+ 11171us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.439060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.439163 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.443307 (+ 4144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.443312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.443337 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:16:20.443508 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.443521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.445415 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.445422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.447245 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.447260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.449066 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.449072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.449257 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.449291 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.449353 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.449492 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"thread_start_us":11868,"threads_started":1}
I20260927 22:16:20.451016 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.428500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.23.173.65:37853)
0927 22:16:20.442699 (+ 14199us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.442708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.442738 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:16:20.442841 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.442849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.443098 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.443319 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:16:20.443710 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.445252 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.447438 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.448894 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.450248 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.450472 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.450537 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.450692 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14023,"thread_start_us":13900,"threads_started":1}
I20260927 22:16:20.452652 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.426182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:43123 (local address 127.0.0.1:50666)
0927 22:16:20.427734 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.427754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.427868 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.444482 (+ 16614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.444487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.444512 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:16:20.444851 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.444862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.447983 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.447988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.449947 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.449984 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.451870 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.451878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.452062 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.452100 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.452171 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.452311 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"thread_start_us":2255,"threads_started":1}
I20260927 22:16:20.453898 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.426707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.23.173.66:43123)
0927 22:16:20.443075 (+ 16368us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.443081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.443103 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:16:20.443180 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.443185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.444263 (+ 1078us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.445023 (+ 760us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:16:20.445245 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.446906 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.450184 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.451638 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.453137 (+ 1499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.453371 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.453436 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.453593 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16208,"thread_start_us":16099,"threads_started":1}
I20260927 22:16:20.454824 2240 tablet_service.cc:1467] Processing CreateTablet for tablet c1a540bbd066488daab003affd34e088 (DEFAULT_TABLE table=test-table [id=cfc793943d874622a8fa54e6f7f11aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:16:20.456282 2240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a540bbd066488daab003affd34e088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:20.458705 2374 tablet_service.cc:1467] Processing CreateTablet for tablet c1a540bbd066488daab003affd34e088 (DEFAULT_TABLE table=test-table [id=cfc793943d874622a8fa54e6f7f11aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:16:20.459980 2374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a540bbd066488daab003affd34e088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:16:20.471863 2590 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Bootstrap starting.
I20260927 22:16:20.477903 2591 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Bootstrap starting.
I20260927 22:16:20.478446 2590 tablet_bootstrap.cc:654] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:20.479912 2590 log.cc:824] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:20.480017 2592 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Bootstrap starting.
I20260927 22:16:20.482981 2591 tablet_bootstrap.cc:654] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:20.483680 2592 tablet_bootstrap.cc:654] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Neither blocks nor log segments found. Creating new log.
I20260927 22:16:20.485103 2592 log.cc:824] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:20.485354 2591 log.cc:824] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:16:20.489522 2590 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: No bootstrap required, opened a new log
I20260927 22:16:20.489997 2590 ts_tablet_manager.cc:1397] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260927 22:16:20.494620 2592 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: No bootstrap required, opened a new log
I20260927 22:16:20.494962 2592 ts_tablet_manager.cc:1397] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260927 22:16:20.495812 2591 tablet_bootstrap.cc:492] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: No bootstrap required, opened a new log
I20260927 22:16:20.496107 2591 ts_tablet_manager.cc:1397] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260927 22:16:20.503399 2590 raft_consensus.cc:348] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.504014 2590 raft_consensus.cc:374] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:20.504172 2590 raft_consensus.cc:729] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2173d95d5d46466398539d46dc84fe11, State: Initialized, Role: FOLLOWER
I20260927 22:16:20.505069 2590 consensus_queue.cc:260] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.505915 2592 raft_consensus.cc:348] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.506465 2592 raft_consensus.cc:374] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:20.506598 2592 raft_consensus.cc:729] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea07b36d625a4cae928fa514c0955727, State: Initialized, Role: FOLLOWER
I20260927 22:16:20.507333 2592 consensus_queue.cc:260] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.507812 2591 raft_consensus.cc:348] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.508387 2591 raft_consensus.cc:374] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:16:20.508543 2591 raft_consensus.cc:729] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df96cb47a39458d91638e11cb30739c, State: Initialized, Role: FOLLOWER
I20260927 22:16:20.509419 2591 consensus_queue.cc:260] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.516181 2592 ts_tablet_manager.cc:1428] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260927 22:16:20.516718 2439 heartbeater.cc:503] Master 127.23.173.126:34245 was elected leader, sending a full tablet report...
I20260927 22:16:20.518692 2572 heartbeater.cc:503] Master 127.23.173.126:34245 was elected leader, sending a full tablet report...
I20260927 22:16:20.519426 2590 ts_tablet_manager.cc:1428] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s
I20260927 22:16:20.525655 2591 ts_tablet_manager.cc:1428] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Time spent starting tablet: real 0.029s user 0.013s sys 0.013s
I20260927 22:16:20.559252 2596 raft_consensus.cc:482] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:16:20.559558 2596 raft_consensus.cc:504] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.561213 2596 leader_election.cc:288] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea07b36d625a4cae928fa514c0955727 (127.23.173.66:43123), 8df96cb47a39458d91638e11cb30739c (127.23.173.65:37853)
I20260927 22:16:20.567901 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.561849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:60005 (local address 127.23.173.65:37853)
0927 22:16:20.562070 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.562078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.562100 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:16:20.562191 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.562196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.562281 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.562383 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:16:20.563042 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.564051 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.565514 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.566807 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.566955 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.567024 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.567506 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.567645 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:16:20.567979 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.561469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:37853 (local address 127.23.173.67:60005)
0927 22:16:20.561802 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.561835 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.561975 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.562557 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.562562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.562593 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:16:20.562836 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.562849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.564203 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.564207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.565367 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.565378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.566983 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.566991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.567276 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.567329 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.567451 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.567623 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 22:16:20.569119 2260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a540bbd066488daab003affd34e088" candidate_uuid: "2173d95d5d46466398539d46dc84fe11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df96cb47a39458d91638e11cb30739c" is_pre_election: true
I20260927 22:16:20.569741 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.562433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:51709 (local address 127.23.173.66:43123)
0927 22:16:20.562623 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.562631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.563245 (+ 614us) server_negotiation.cc:407] Connection header received
0927 22:16:20.563422 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.563430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.563533 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.563637 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:16:20.564167 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.565467 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.567633 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.568882 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.569056 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.569124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.569374 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.569497 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:16:20.569836 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.561699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:43123 (local address 127.23.173.67:51709)
0927 22:16:20.563134 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0927 22:16:20.563153 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:16:20.563267 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:16:20.563659 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:16:20.563663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:16:20.563688 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:16:20.564005 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.564015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.565623 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.565629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.567466 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:16:20.567481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.569063 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.569069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:16:20.569225 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.569246 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:16:20.569353 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:16:20.569470 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"mutex_wait_us":6,"thread_start_us":1095,"threads_started":1}
I20260927 22:16:20.569761 2260 raft_consensus.cc:2454] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2173d95d5d46466398539d46dc84fe11 in term 0.
I20260927 22:16:20.570549 2462 leader_election.cc:302] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2173d95d5d46466398539d46dc84fe11, 8df96cb47a39458d91638e11cb30739c; no voters:
I20260927 22:16:20.570917 2596 raft_consensus.cc:2781] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:16:20.571072 2596 raft_consensus.cc:482] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:16:20.571170 2596 raft_consensus.cc:3037] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:20.570991 2394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a540bbd066488daab003affd34e088" candidate_uuid: "2173d95d5d46466398539d46dc84fe11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea07b36d625a4cae928fa514c0955727" is_pre_election: true
I20260927 22:16:20.571642 2394 raft_consensus.cc:2454] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2173d95d5d46466398539d46dc84fe11 in term 0.
I20260927 22:16:20.573572 2596 raft_consensus.cc:504] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.574266 2596 leader_election.cc:288] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [CANDIDATE]: Term 1 election: Requested vote from peers ea07b36d625a4cae928fa514c0955727 (127.23.173.66:43123), 8df96cb47a39458d91638e11cb30739c (127.23.173.65:37853)
I20260927 22:16:20.574759 2394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a540bbd066488daab003affd34e088" candidate_uuid: "2173d95d5d46466398539d46dc84fe11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea07b36d625a4cae928fa514c0955727"
I20260927 22:16:20.574932 2260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a540bbd066488daab003affd34e088" candidate_uuid: "2173d95d5d46466398539d46dc84fe11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df96cb47a39458d91638e11cb30739c"
I20260927 22:16:20.575006 2394 raft_consensus.cc:3037] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:20.575186 2260 raft_consensus.cc:3037] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:16:20.578680 2260 raft_consensus.cc:2454] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2173d95d5d46466398539d46dc84fe11 in term 1.
I20260927 22:16:20.579236 2462 leader_election.cc:302] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2173d95d5d46466398539d46dc84fe11, 8df96cb47a39458d91638e11cb30739c; no voters:
I20260927 22:16:20.579388 2394 raft_consensus.cc:2454] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2173d95d5d46466398539d46dc84fe11 in term 1.
I20260927 22:16:20.579515 2596 raft_consensus.cc:2781] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:16:20.580186 2596 raft_consensus.cc:686] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 1 LEADER]: Becoming Leader. State: Replica: 2173d95d5d46466398539d46dc84fe11, State: Running, Role: LEADER
I20260927 22:16:20.580792 2596 consensus_queue.cc:237] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } }
I20260927 22:16:20.586500 2116 catalog_manager.cc:5495] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2173d95d5d46466398539d46dc84fe11 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "2173d95d5d46466398539d46dc84fe11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2173d95d5d46466398539d46dc84fe11" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 35073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 } health_report { overall_health: UNKNOWN } } }
W20260927 22:16:20.590606 2440 tablet.cc:2369] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:16:20.592670 2306 tablet.cc:2369] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:16:20.597265 2573 tablet.cc:2369] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:16:20.628237 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.621715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.23.173.67:35073)
0927 22:16:20.621970 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.621979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.622001 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:16:20.622117 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.622125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.622279 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.622449 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:16:20.622750 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.624061 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.625317 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.626207 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.626361 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.626831 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:16:20.627562 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:16:20.627737 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.627791 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.627888 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:16:20.637764 2394 raft_consensus.cc:1261] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 1 FOLLOWER]: Refusing update from remote peer 2173d95d5d46466398539d46dc84fe11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:16:20.637759 2260 raft_consensus.cc:1261] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 1 FOLLOWER]: Refusing update from remote peer 2173d95d5d46466398539d46dc84fe11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:16:20.638825 2600 consensus_queue.cc:1035] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df96cb47a39458d91638e11cb30739c" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:16:20.639181 2596 consensus_queue.cc:1035] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea07b36d625a4cae928fa514c0955727" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:16:20.704178 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:16:20.704523 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:16:20.706107 2606 mvcc.cc:205] Tried to move back new op lower bound from 7334082071081631744 to 7334082070863118336. Current Snapshot: MvccSnapshot[applied={T|T < 7334082071081631744 or (T in {7334082071081631744})}]
I20260927 22:16:20.709652 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29f656c2f3754a619b24ecb09e943309: Sending DeleteTablet for 3 replicas of tablet c1a540bbd066488daab003affd34e088
I20260927 22:16:20.710920 2507 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a540bbd066488daab003affd34e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:16:20 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:46580
I20260927 22:16:20.711071 2240 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a540bbd066488daab003affd34e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:16:20 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:44982
I20260927 22:16:20.712085 2374 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a540bbd066488daab003affd34e088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:16:20 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:50666
I20260927 22:16:20.713114 2615 tablet_replica.cc:330] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: stopping tablet replica
I20260927 22:16:20.713358 2613 tablet_replica.cc:330] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: stopping tablet replica
I20260927 22:16:20.713625 2615 raft_consensus.cc:2229] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:16:20.713784 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 2177
I20260927 22:16:20.714238 2615 raft_consensus.cc:2258] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:16:20.714195 2613 raft_consensus.cc:2229] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:16:20.714847 2613 raft_consensus.cc:2258] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:16:20.715353 2614 tablet_replica.cc:330] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: stopping tablet replica
I20260927 22:16:20.715739 2614 raft_consensus.cc:2229] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:16:20.716238 2614 raft_consensus.cc:2258] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:16:20.716658 2615 ts_tablet_manager.cc:1905] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:16:20.719481 2613 ts_tablet_manager.cc:1905] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:16:20.721227 2614 ts_tablet_manager.cc:1905] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:16:20.724466 2615 ts_tablet_manager.cc:1918] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:16:20.724721 2615 log.cc:1196] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/c1a540bbd066488daab003affd34e088
I20260927 22:16:20.725364 2615 ts_tablet_manager.cc:1939] T c1a540bbd066488daab003affd34e088 P ea07b36d625a4cae928fa514c0955727: Deleting consensus metadata
I20260927 22:16:20.727625 2101 catalog_manager.cc:4841] TS ea07b36d625a4cae928fa514c0955727 (127.23.173.66:43123): tablet c1a540bbd066488daab003affd34e088 (table test-table [id=cfc793943d874622a8fa54e6f7f11aaf]) successfully deleted
Sep 27 22:16:20 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:16:20.729471 2613 ts_tablet_manager.cc:1918] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:16:20.729650 2613 log.cc:1196] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/c1a540bbd066488daab003affd34e088
I20260927 22:16:20.730312 2613 ts_tablet_manager.cc:1939] T c1a540bbd066488daab003affd34e088 P 2173d95d5d46466398539d46dc84fe11: Deleting consensus metadata
I20260927 22:16:20.732437 2102 catalog_manager.cc:4841] TS 2173d95d5d46466398539d46dc84fe11 (127.23.173.67:35073): tablet c1a540bbd066488daab003affd34e088 (table test-table [id=cfc793943d874622a8fa54e6f7f11aaf]) successfully deleted
I20260927 22:16:20.734705 2614 ts_tablet_manager.cc:1918] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:16:20.736372 2614 log.cc:1196] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/c1a540bbd066488daab003affd34e088
I20260927 22:16:20.737248 2614 ts_tablet_manager.cc:1939] T c1a540bbd066488daab003affd34e088 P 8df96cb47a39458d91638e11cb30739c: Deleting consensus metadata
I20260927 22:16:20.739171 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.717281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.23.173.65:37853)
0927 22:16:20.717479 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.717486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.717526 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:16:20.717591 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.717597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.717682 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.717794 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:20.719398 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.720543 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.722789 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.723454 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.723552 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.729434 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:20.729447 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:20.729465 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:20.729510 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:20.735481 (+ 5971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.736405 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.736415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.736418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.736582 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.737048 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.737053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.737056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.737610 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:20.737906 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.738549 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.738774 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:16:20.739359 2102 catalog_manager.cc:4841] TS 8df96cb47a39458d91638e11cb30739c (127.23.173.65:37853): tablet c1a540bbd066488daab003affd34e088 (table test-table [id=cfc793943d874622a8fa54e6f7f11aaf]) successfully deleted
I20260927 22:16:20.740840 2300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:20.890697 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 2177
I20260927 22:16:20.907995 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 2311
Sep 27 22:16:20 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:16:20.924472 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:20.908834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.23.173.66:43123)
0927 22:16:20.909051 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:20.909058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:20.909080 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:16:20.909149 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:20.909155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:20.909239 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:20.909362 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:20.910733 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.911922 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.913062 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:20.913698 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:20.913826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:20.917804 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:20.917820 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:20.917840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:20.917893 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:20.921588 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.922438 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.922449 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.922453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.922607 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:20.922930 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:20.922935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:20.922939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:20.923407 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:20.923685 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:20.923876 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:16:20.924092 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:16:20.925283 2434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:21.012756 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 2311
I20260927 22:16:21.030222 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 2444
Sep 27 22:16:21 dist-test-slave-3wc3 krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:16:21.046566 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:16:21.031049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.23.173.67:35073)
0927 22:16:21.031279 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:16:21.031284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:16:21.031302 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:16:21.031466 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:16:21.031470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:16:21.031540 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:16:21.031653 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:16:21.032889 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:21.033697 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:21.035108 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:16:21.035632 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:16:21.035768 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:16:21.040619 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:16:21.040631 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:16:21.040645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:16:21.040683 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:16:21.043750 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:21.044401 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:21.044408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:21.044410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:21.044556 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:16:21.044949 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:16:21.044955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:16:21.044958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:16:21.045390 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:16:21.045646 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:16:21.046047 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:16:21.046217 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 22:16:21.047279 2567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:21.167888 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 2444
I20260927 22:16:21.187346 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 2084
I20260927 22:16:21.188522 2146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:16:21.296502 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 2084
2026-09-27T22:16:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4382 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:16:21 dist-test-slave-3wc3 krb5kdc[2623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:16:21 dist-test-slave-3wc3 krb5kdc[2623](info): set up 2 sockets
Sep 27 22:16:21 dist-test-slave-3wc3 krb5kdc[2623](info): commencing operation
krb5kdc: starting...
W20260927 22:16:22.398864 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:16:22 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:16:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangeradmin_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangeradmin_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangerlookup_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangerlookup_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
I20260927 22:16:22.708943 24245 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskO6uUAX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/postgres -l logfile start
2026-09-27 22:16:25.165 UTC [2653] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:16:25.165 UTC [2653] LOG: listening on IPv4 address "127.23.173.84", port 32967
2026-09-27 22:16:25.168 UTC [2653] LOG: listening on Unix socket "/tmp/.s.PGSQL.32967"
2026-09-27 22:16:25.190 UTC [2656] LOG: database system was shut down at 2026-09-27 22:16:23 UTC
2026-09-27 22:16:25.193 UTC [2653] LOG: database system is ready to accept connections
I20260927 22:16:26.170445 24245 mini_postgres.cc:96] Postgres bound to 32967
2026-09-27 22:16:26.189 UTC [2664] FATAL: database "slave" does not exist
127.23.173.84:32967 - accepting connections
I20260927 22:16:26.190042 24245 mini_ranger.cc:161] Starting Ranger...
I20260927 22:16:26.214718 24245 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:16:26.697263 24245 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:16:26.697372 24245 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin
2026-09-27 22:16:26,984 [I] DB FLAVOR :POSTGRES
2026-09-27 22:16:26,984 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:16:26,985 [I] Checking connection..
2026-09-27 22:16:26,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:16:27,240 [I] Checking connection passed.
2026-09-27 22:16:27,240 [I] --------- Verifying version history table ---------
2026-09-27 22:16:27,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:16:27,463 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:16:27,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:16:27,696 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:16:27,696 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:16:27,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:16:27.952 UTC [2728] WARNING: there is no transaction in progress
2026-09-27 22:16:27,958 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:16:27,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:16:28,199 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:16:28,199 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:16:28,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:16:28,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:16:28,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3wc3.c.gcp-upstream.internal','N') ;"
2026-09-27 22:16:28,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:16:29,208 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:16:29,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:16:29,431 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:16:29,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:16:29,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:16:29,931 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:16:29,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:16:30.876 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:30.892 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:30.910 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:30.925 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:30.941 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:30.961 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.014 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.020 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.034 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.044 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.054 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.068 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.077 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.083 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.089 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31.095 UTC [2836] WARNING: there is no transaction in progress
2026-09-27 22:16:31,715 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:16:31,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:16:31,948 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:16:31,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:16:32,195 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:16:32,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:16:32,451 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:16:32,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:16:32,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:16:32,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3wc3.c.gcp-upstream.internal';"
2026-09-27 22:16:33,194 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:16:33,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:16:33,414 [I] DB_PATCHES have already been applied
I20260927 22:16:33.419860 24245 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskO6uUAX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:16:33.419950 24245 mini_ranger.cc:193] Using host: 127.23.173.84
I20260927 22:16:33.434881 24245 mini_ranger.cc:227] Ranger admin URL: http://127.23.173.84:48871
Sep 27, 2026 10:16:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:16:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:16:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:16:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:16:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:16:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:16:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:16:34.574251 24245 mini_ranger.cc:160] Time spent starting Ranger: real 8.384s user 0.007s sys 0.031s
Sep 27, 2026 10:16:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:16:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 27, 2026 10:16:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:16:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:16:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:16:56.198743 24245 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangerkms_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/rangerkms_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.173.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:16:56 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547416, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:16:56.373628 24245 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:16:56.394156 24245 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:16:57.242182 24245 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:16:57.242421 24245 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms
I20260927 22:16:57.242659 24245 mini_ranger_kms.cc:227] Using postgres at 127.23.173.84:32967
2026-09-27 22:16:57,694 [I] DB FLAVOR :POSTGRES
2026-09-27 22:16:57,694 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:16:57,695 [I] Checking connection
2026-09-27 22:16:57,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:16:57,921 [I] connection success
2026-09-27 22:16:57,922 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:16:57,922 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:16:57,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:16:58,179 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:16:58,180 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:16:58,180 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:16:58,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:32967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:16:58,454 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:16:58.905161 24245 mini_ranger_kms.cc:310] Created kms service
I20260927 22:16:59.104959 24245 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:16:59.147357 24245 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:16:59 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547416, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:16:59.578087 24245 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:16:59.578526 24245 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms:/tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:16:59.578599 24245 mini_ranger_kms.cc:240] Using host: 127.23.173.84
I20260927 22:16:59.600121 24245 mini_ranger_kms.cc:276] Ranger KMS PID: 3016
I20260927 22:16:59.600332 24245 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.173.84:38373
Sep 27, 2026 10:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:17:01.872962 24245 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.499s user 0.104s sys 0.374s
I20260927 22:17:01.873299 24245 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:17:01.873529 24245 mini_ranger_kms.cc:401] 127.23.173.84:38373/kms/v1/keys
Sep 27 22:17:01 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:17:02 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
Sep 27 22:17:02 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
Sep 27 22:17:04 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:17:05.014324 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38979
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:38979
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:38373/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:17:05.515913 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:05.516347 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:05.516558 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:05.524784 3067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:17:05.524933 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:05.525027 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:05.525166 3067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:17:05.525260 3067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:17:05.536617 3067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.173.84:38373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:38979
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:05.538298 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:05.539716 3067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:05.554939 3072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:05.554836 3073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:05.554803 3075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:05.555595 3067 server_base.cc:1034] running on GCE node
I20260927 22:17:05.556381 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:05.558200 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:05.559424 3067 hybrid_clock.cc:648] HybridClock initialized: now 1790547425559358 us; error 71 us; skew 500 ppm
I20260927 22:17:05.559875 3067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:05 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:05.567173 3067 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:17:05.568831 3067 webserver.cc:460] Webserver started at http://127.23.173.126:43895/ using document root <none> and password file <none>
I20260927 22:17:05.569561 3067 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:05.569659 3067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:05.570052 3067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:05 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:05.625982 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "1ad2a0c1007f46cfa3e5e315f771736f"
format_stamp: "Formatted at 2026-09-27 22:17:05 on dist-test-slave-3wc3"
server_key: "f048460df94265eafe96d8180aadee08"
server_key_iv: "0d1298146dbe58246664d19f24eeead8"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:05.626889 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "1ad2a0c1007f46cfa3e5e315f771736f"
format_stamp: "Formatted at 2026-09-27 22:17:05 on dist-test-slave-3wc3"
server_key: "f048460df94265eafe96d8180aadee08"
server_key_iv: "0d1298146dbe58246664d19f24eeead8"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:05.633309 3067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:17:05.654680 3082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:05.656787 3067 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:17:05.656999 3067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "1ad2a0c1007f46cfa3e5e315f771736f"
format_stamp: "Formatted at 2026-09-27 22:17:05 on dist-test-slave-3wc3"
server_key: "f048460df94265eafe96d8180aadee08"
server_key_iv: "0d1298146dbe58246664d19f24eeead8"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:05.657361 3067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:05.705154 3067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:05.709244 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:05.709520 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:05.729235 3067 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:38979
I20260927 22:17:05.729259 3133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:38979 every 8 connection(s)
I20260927 22:17:05.731304 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:17:05.737254 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:17:05.737876 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 3067
I20260927 22:17:05.738137 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:17:05.739256 24245 external_mini_cluster.cc:1428] Setting key da626c27d3684fc0d4bcf2322087c422
I20260927 22:17:05.754623 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: Bootstrap starting.
Sep 27 22:17:05 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:05.758433 3134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: Neither blocks nor log segments found. Creating new log.
I20260927 22:17:05.759636 3134 log.cc:824] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:05.762833 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: No bootstrap required, opened a new log
I20260927 22:17:05.764231 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:05.742297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.23.173.126:38979)
0927 22:17:05.743132 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:05.743144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:05.743184 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:17:05.745043 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:05.745065 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:05.745519 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:05.745863 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:05.747616 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:05.748947 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:05.750988 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:05.751697 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:05.751844 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:05.756598 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:05.756608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:05.756623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:05.756662 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:05.760052 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:05.760756 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:05.760762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:05.760769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:05.760909 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:05.761323 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:05.761332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:05.761334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:05.761904 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:05.762433 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:05.762847 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:17:05.763279 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":364,"threads_started":1}
I20260927 22:17:05.771201 3134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } }
I20260927 22:17:05.771708 3134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:17:05.771864 3134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad2a0c1007f46cfa3e5e315f771736f, State: Initialized, Role: FOLLOWER
I20260927 22:17:05.772667 3134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } }
I20260927 22:17:05.772881 3134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:17:05.772981 3134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:17:05.773103 3134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:17:05.775303 3134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } }
I20260927 22:17:05.775739 3134 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad2a0c1007f46cfa3e5e315f771736f; no voters:
I20260927 22:17:05.776257 3134 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:17:05.776401 3139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:17:05.777326 3139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [term 1 LEADER]: Becoming Leader. State: Replica: 1ad2a0c1007f46cfa3e5e315f771736f, State: Running, Role: LEADER
I20260927 22:17:05.778105 3139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } }
I20260927 22:17:05.779126 3134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:17:05.782038 3140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } } }
I20260927 22:17:05.782002 3141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad2a0c1007f46cfa3e5e315f771736f. Latest consensus state: current_term: 1 leader_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad2a0c1007f46cfa3e5e315f771736f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38979 } } }
I20260927 22:17:05.782606 3140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:05.782606 3141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:05.784673 3145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:17:05.791337 3145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:17:05.792317 3134 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:17:05.811681 3145 catalog_manager.cc:1347] Generated new cluster ID: ccad551971b1405cb9857513a6ac57fb
I20260927 22:17:05.811968 3145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:17:05.842710 3145 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:17:05.844776 3145 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:17:05.856117 3145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: Generated new TSK 0
I20260927 22:17:05.857287 3145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:17:06.602728 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:38373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:06 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547426, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:17:07.005900 3179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:07.006251 3179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:07.006397 3179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:07.013760 3179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:07.013908 3179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:07.014124 3179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:17:07.024513 3179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.173.84:38373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:07.026041 3179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:07.027534 3179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:07.042589 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:07.043128 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:07.043633 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:07.044317 3179 server_base.cc:1034] running on GCE node
I20260927 22:17:07.045070 3179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:07.047482 3179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:07.048756 3179 hybrid_clock.cc:648] HybridClock initialized: now 1790547427048704 us; error 45 us; skew 500 ppm
I20260927 22:17:07.049166 3179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:07.052654 3179 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:17:07.054380 3179 webserver.cc:460] Webserver started at http://127.23.173.65:38763/ using document root <none> and password file <none>
I20260927 22:17:07.055328 3179 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:07.055514 3179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:07.055909 3179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:07.083038 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "76faf2676b1645fb8d1d1b6859dabb0f"
format_stamp: "Formatted at 2026-09-27 22:17:07 on dist-test-slave-3wc3"
server_key: "67f33f77e13d8b188b224397de4c0a96"
server_key_iv: "10c559a4657fac2396cf6735a4e06647"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:07.084151 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "76faf2676b1645fb8d1d1b6859dabb0f"
format_stamp: "Formatted at 2026-09-27 22:17:07 on dist-test-slave-3wc3"
server_key: "67f33f77e13d8b188b224397de4c0a96"
server_key_iv: "10c559a4657fac2396cf6735a4e06647"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:07.099961 3179 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.008s
I20260927 22:17:07.126649 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:07.128674 3179 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.004s
I20260927 22:17:07.128875 3179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "76faf2676b1645fb8d1d1b6859dabb0f"
format_stamp: "Formatted at 2026-09-27 22:17:07 on dist-test-slave-3wc3"
server_key: "67f33f77e13d8b188b224397de4c0a96"
server_key_iv: "10c559a4657fac2396cf6735a4e06647"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:07.129132 3179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:07.162241 3179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:07.166816 3179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:07.167153 3179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:07.168255 3179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:07.169938 3179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:07.170097 3179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 22:17:07.170295 3179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:07.170372 3179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:07.228641 3179 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:39005
I20260927 22:17:07.228691 3311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:39005 every 8 connection(s)
I20260927 22:17:07.230412 3179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:17:07.236760 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 3179
I20260927 22:17:07.236969 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:17:07.237797 24245 external_mini_cluster.cc:1428] Setting key 4dd9155dcb17a132a10869bdf46620bc
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:07.268590 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:07.237280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38979 (local address 127.23.173.65:33283)
0927 22:17:07.239453 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:07.239504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:07.241904 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:07.243296 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:07.243310 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:07.243987 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:07.244870 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:07.244899 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:07.246656 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:07.246661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:07.248733 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:07.248769 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:07.250068 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:07.250076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:07.250374 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:07.251208 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:07.251239 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:07.254039 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:07.258151 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:07.258161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:07.258180 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:07.258790 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:07.259233 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:07.259238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:07.259242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:07.259480 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:07.262547 (+ 3067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:07.262556 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:07.264923 (+ 2367us) client_negotiation.cc:769] Sending connection context
0927 22:17:07.265258 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 22:17:07.265854 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1829,"thread_start_us":1585,"threads_started":1}
I20260927 22:17:07.271665 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:07.236515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:33283 (local address 127.23.173.126:38979)
0927 22:17:07.242669 (+ 6154us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:07.242676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:07.242695 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:17:07.242754 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:07.242760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:07.242853 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:07.245064 (+ 2211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:07.245238 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:07.246432 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:07.248933 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:07.249590 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:07.249716 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:07.254315 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:07.254322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:07.254327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:07.254368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:07.257933 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:07.258933 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:07.258938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:07.258941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:07.259098 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:07.261772 (+ 2674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:07.261796 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:07.261800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:07.262270 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:07.262447 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:07.269441 (+ 6994us) server_negotiation.cc:299] Negotiation successful
0927 22:17:07.269735 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6008,"thread_start_us":13346,"threads_started":1}
I20260927 22:17:07.273547 3312 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38979
I20260927 22:17:07.274021 3312 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:07.275157 3312 heartbeater.cc:511] Master 127.23.173.126:38979 requested a full tablet report, sending...
I20260927 22:17:07.278707 3099 ts_manager.cc:194] Registered new tserver with Master: 76faf2676b1645fb8d1d1b6859dabb0f (127.23.173.65:39005)
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:17:07.280881 3099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:33283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:17:07.317903 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:38373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547426, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547426, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
W20260927 22:17:07.862443 3318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:07.862948 3318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:07.863176 3318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:07.874739 3318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:07.874966 3318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:07.875207 3318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:17:07.890887 3318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.173.84:38373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:07.892915 3318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:07.894781 3318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:07.931672 3325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:07.932657 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:07.933175 3318 server_base.cc:1034] running on GCE node
W20260927 22:17:07.933706 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:07.934602 3318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:07.941394 3318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:07.942742 3318 hybrid_clock.cc:648] HybridClock initialized: now 1790547427942639 us; error 95 us; skew 500 ppm
I20260927 22:17:07.943198 3318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:07.947446 3318 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:17:07.980909 3318 webserver.cc:460] Webserver started at http://127.23.173.66:43925/ using document root <none> and password file <none>
I20260927 22:17:07.982089 3318 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:07.982323 3318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:07.982779 3318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:07 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:08.014750 3318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "7209bc363c654b78825d4d84c21f6ba9"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "ceba32ce0f72e3bdd2ff32f8eb25fcd8"
server_key_iv: "85fd9490ae6ea4ae38025c2a80ea75fd"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.016151 3318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "7209bc363c654b78825d4d84c21f6ba9"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "ceba32ce0f72e3bdd2ff32f8eb25fcd8"
server_key_iv: "85fd9490ae6ea4ae38025c2a80ea75fd"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.023582 3318 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260927 22:17:08.045501 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.047374 3318 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:17:08.047619 3318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "7209bc363c654b78825d4d84c21f6ba9"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "ceba32ce0f72e3bdd2ff32f8eb25fcd8"
server_key_iv: "85fd9490ae6ea4ae38025c2a80ea75fd"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.047938 3318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:08.085911 3318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:08.091084 3318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:08.091413 3318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:08.092622 3318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:08.094399 3318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:08.094511 3318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.094683 3318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:08.094746 3318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.138172 3318 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:39553
I20260927 22:17:08.138196 3454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:39553 every 8 connection(s)
I20260927 22:17:08.139993 3318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:17:08.144543 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 3318
I20260927 22:17:08.144708 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:17:08.145251 24245 external_mini_cluster.cc:1428] Setting key e49018e42558c997f8d518d2c10fd6f2
Sep 27 22:17:08 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
I20260927 22:17:08.182789 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.147377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:54715 (local address 127.23.173.126:38979)
0927 22:17:08.160974 (+ 13597us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.160981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.162224 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 22:17:08.164324 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.164332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.164434 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.164527 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:08.166801 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.167874 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.170188 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.170806 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.170898 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.175503 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:08.175512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:08.175516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:08.175558 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:08.179030 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.179951 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.179956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.179960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.180102 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.180616 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.180621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.180625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.180995 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:08.181122 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.182125 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.182357 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13458,"thread_start_us":13344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
I20260927 22:17:08.183928 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.143729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38979 (local address 127.23.173.66:54715)
0927 22:17:08.161571 (+ 17842us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:08.161633 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:08.163959 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:08.165104 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:08.165116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:08.165760 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:08.166607 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.166631 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.168078 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.168083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.170035 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.170049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.171225 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.171232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.171530 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.172320 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:08.172349 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:08.175242 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:08.179207 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:08.179217 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:08.179235 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:08.179806 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:08.180242 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:08.180247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:08.180250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:08.180493 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:08.181214 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:08.181221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:08.181657 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 22:17:08.181972 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 22:17:08.182529 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17510,"thread_start_us":17277,"threads_started":1}
I20260927 22:17:08.185825 3455 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38979
I20260927 22:17:08.186144 3455 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:08.187013 3455 heartbeater.cc:511] Master 127.23.173.126:38979 requested a full tablet report, sending...
I20260927 22:17:08.188714 3099 ts_manager.cc:194] Registered new tserver with Master: 7209bc363c654b78825d4d84c21f6ba9 (127.23.173.66:39553)
I20260927 22:17:08.189772 3099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:54715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:17:08.217463 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:38373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:17:08.285596 3312 heartbeater.cc:503] Master 127.23.173.126:38979 was elected leader, sending a full tablet report...
W20260927 22:17:08.601447 3461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:08.601706 3461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:08.601814 3461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:08.609061 3461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:08.609186 3461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:08.609376 3461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:17:08.618960 3461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.173.84:38373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:08.620203 3461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:08.621466 3461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:08.634557 3466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:08.634649 3467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:08.634770 3469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:08.635643 3461 server_base.cc:1034] running on GCE node
I20260927 22:17:08.636235 3461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:08.637135 3461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:08.638363 3461 hybrid_clock.cc:648] HybridClock initialized: now 1790547428638316 us; error 39 us; skew 500 ppm
I20260927 22:17:08.638720 3461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:08 dist-test-slave-3wc3 krb5kdc[2623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547428, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:08.641763 3461 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:17:08.643181 3461 webserver.cc:460] Webserver started at http://127.23.173.67:45539/ using document root <none> and password file <none>
I20260927 22:17:08.643813 3461 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:08.643909 3461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:08.644155 3461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:08 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547428, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:08.658733 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "6049ddd7c48c461b93b0953b622df0dd"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "63e8bcadcf5ffc8c84254f4044545639"
server_key_iv: "f3589a9771869ff4816398be4f7014ee"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.659616 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "6049ddd7c48c461b93b0953b622df0dd"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "63e8bcadcf5ffc8c84254f4044545639"
server_key_iv: "f3589a9771869ff4816398be4f7014ee"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.665160 3461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:17:08.682565 3476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.684309 3461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 22:17:08.684485 3461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "6049ddd7c48c461b93b0953b622df0dd"
format_stamp: "Formatted at 2026-09-27 22:17:08 on dist-test-slave-3wc3"
server_key: "63e8bcadcf5ffc8c84254f4044545639"
server_key_iv: "f3589a9771869ff4816398be4f7014ee"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:08.684746 3461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:08.702916 3461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:08.706512 3461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:08.706773 3461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:08.707726 3461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:08.709028 3461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:08.709100 3461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.709185 3461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:08.709271 3461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:08.742914 3461 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:42519
I20260927 22:17:08.742950 3588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:42519 every 8 connection(s)
I20260927 22:17:08.744210 3461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:17:08.748373 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 3461
I20260927 22:17:08.748556 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:17:08.749008 24245 external_mini_cluster.cc:1428] Setting key 49c29687e575d6a6ae0f656a6e7e7c13
Sep 27 22:17:08 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547428, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:08.767637 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.746705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:55351 (local address 127.23.173.126:38979)
0927 22:17:08.747197 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.747202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.747990 (+ 788us) server_negotiation.cc:407] Connection header received
0927 22:17:08.749869 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.749874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.750011 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.750158 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:08.751678 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.752586 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.754445 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.754955 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.755053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.761826 (+ 6773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:08.761832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:08.761835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:08.761879 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:08.764949 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.765631 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.765635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.765637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.765758 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.766200 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.766203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.766205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.766468 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:08.766580 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.767220 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.767391 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
I20260927 22:17:08.768623 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.747185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38979 (local address 127.23.173.67:55351)
0927 22:17:08.747831 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:08.747873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:08.749573 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:08.750408 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:08.750426 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:08.750837 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:08.751511 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.751527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.752714 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.752718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.754280 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.754299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.755144 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.755148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.755344 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.759568 (+ 4224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:08.759590 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:08.761618 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:08.765084 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:08.765091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:08.765105 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:08.765532 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:08.765847 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:08.765851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:08.765853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:08.766109 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:08.766575 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:08.766581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:08.766894 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 22:17:08.767128 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 22:17:08.767502 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":256,"threads_started":1}
I20260927 22:17:08.770026 3589 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38979
I20260927 22:17:08.770324 3589 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:08.771021 3589 heartbeater.cc:511] Master 127.23.173.126:38979 requested a full tablet report, sending...
I20260927 22:17:08.772419 3099 ts_manager.cc:194] Registered new tserver with Master: 6049ddd7c48c461b93b0953b622df0dd (127.23.173.67:42519)
I20260927 22:17:08.773267 3099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:55351
I20260927 22:17:08.776906 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:17:08.794895 3591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.782545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.23.173.126:38979)
0927 22:17:08.782716 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.782720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.783208 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:17:08.783345 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.783349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.783425 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.783507 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:08.784875 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.785602 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.787093 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.787541 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.787673 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.790045 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:08.790050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:08.790054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:08.790088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:08.792670 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.793279 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.793283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.793285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.793407 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:08.793709 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:08.793712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:08.793714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:08.794057 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:08.794235 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.794495 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.794661 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:17:08.799693 3099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:17:08.806886 3099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:17:08.846261 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.821165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.23.173.66:39553)
0927 22:17:08.835738 (+ 14573us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.835749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.838057 (+ 2308us) server_negotiation.cc:407] Connection header received
0927 22:17:08.838424 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.838433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.838724 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.838945 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:17:08.840059 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841091 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.843856 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.845044 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.845204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.845414 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.845712 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.845870 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14352,"thread_start_us":14586,"threads_started":1}
I20260927 22:17:08.846274 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.820974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:39553 (local address 127.0.0.1:56638)
0927 22:17:08.837843 (+ 16869us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:08.837872 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:08.838211 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:08.839293 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:08.839297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:08.839319 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:17:08.839844 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.839853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841247 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.841254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.843698 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.843714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.845245 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.845251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.845424 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.845500 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 22:17:08.845660 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:17:08.845855 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16689,"thread_start_us":16501,"threads_started":1}
I20260927 22:17:08.846951 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.820596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:42519 (local address 127.0.0.1:41066)
0927 22:17:08.822641 (+ 2045us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:08.822692 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:08.822881 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:08.839045 (+ 16164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:08.839053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:08.839082 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:17:08.839832 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.839845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841880 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.841886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.844239 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.844250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.846081 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.846087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.846273 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.846314 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:17:08.846443 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:17:08.846578 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"mutex_wait_us":296,"thread_start_us":1496,"threads_started":1}
I20260927 22:17:08.847329 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.820881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.23.173.67:42519)
0927 22:17:08.838523 (+ 17642us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.838529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.838556 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:17:08.838625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.838630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.838799 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.839029 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:17:08.840140 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841718 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.844380 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.845847 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.846037 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.846267 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.846784 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.846962 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17463,"thread_start_us":17316,"threads_started":1}
I20260927 22:17:08.848407 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.823135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:39005 (local address 127.0.0.1:57198)
0927 22:17:08.838215 (+ 15080us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:08.838235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:08.838365 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:08.839242 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:08.839248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:08.839276 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:17:08.839954 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.839964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841924 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.841932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.844349 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:08.844362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.847615 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.847621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:08.847796 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.847831 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:17:08.847927 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:17:08.848051 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14940,"mutex_wait_us":140,"thread_start_us":14767,"threads_started":1}
I20260927 22:17:08.848521 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:08.823483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.23.173.65:39005)
0927 22:17:08.835213 (+ 11730us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:08.835228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:08.838450 (+ 3222us) server_negotiation.cc:407] Connection header received
0927 22:17:08.838546 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:08.838553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:08.838725 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:08.838945 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:17:08.840189 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.841775 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.844508 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:08.845876 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:08.847224 (+ 1348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:08.847460 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:08.847991 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 22:17:08.848170 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11466,"thread_start_us":10852,"threads_started":1}
I20260927 22:17:08.851039 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 476ba128e57d4639a9790613c38f9610 (DEFAULT_TABLE table=test-table [id=f767e8f1247e4ba99c4510c938e332f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:17:08.851073 3390 tablet_service.cc:1467] Processing CreateTablet for tablet 476ba128e57d4639a9790613c38f9610 (DEFAULT_TABLE table=test-table [id=f767e8f1247e4ba99c4510c938e332f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:17:08.852361 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476ba128e57d4639a9790613c38f9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:17:08.852440 3390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476ba128e57d4639a9790613c38f9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:17:08.852150 3247 tablet_service.cc:1467] Processing CreateTablet for tablet 476ba128e57d4639a9790613c38f9610 (DEFAULT_TABLE table=test-table [id=f767e8f1247e4ba99c4510c938e332f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:17:08.853467 3247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476ba128e57d4639a9790613c38f9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:17:08.864269 3608 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Bootstrap starting.
I20260927 22:17:08.867223 3609 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Bootstrap starting.
I20260927 22:17:08.867606 3608 tablet_bootstrap.cc:654] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Neither blocks nor log segments found. Creating new log.
I20260927 22:17:08.868809 3610 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Bootstrap starting.
I20260927 22:17:08.868963 3608 log.cc:824] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:08.870915 3609 tablet_bootstrap.cc:654] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Neither blocks nor log segments found. Creating new log.
I20260927 22:17:08.872159 3609 log.cc:824] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:08.872462 3610 tablet_bootstrap.cc:654] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Neither blocks nor log segments found. Creating new log.
I20260927 22:17:08.873575 3608 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: No bootstrap required, opened a new log
I20260927 22:17:08.873752 3610 log.cc:824] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:08.873852 3608 ts_tablet_manager.cc:1397] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:17:08.875363 3609 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: No bootstrap required, opened a new log
I20260927 22:17:08.875624 3609 ts_tablet_manager.cc:1397] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:17:08.876360 3610 tablet_bootstrap.cc:492] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: No bootstrap required, opened a new log
I20260927 22:17:08.876672 3610 ts_tablet_manager.cc:1397] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:17:08.886210 3608 raft_consensus.cc:348] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.886879 3608 raft_consensus.cc:374] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:17:08.887030 3608 raft_consensus.cc:729] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6049ddd7c48c461b93b0953b622df0dd, State: Initialized, Role: FOLLOWER
I20260927 22:17:08.887995 3608 consensus_queue.cc:260] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.889158 3609 raft_consensus.cc:348] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.889281 3610 raft_consensus.cc:348] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.889912 3610 raft_consensus.cc:374] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:17:08.890141 3610 raft_consensus.cc:729] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76faf2676b1645fb8d1d1b6859dabb0f, State: Initialized, Role: FOLLOWER
I20260927 22:17:08.890885 3589 heartbeater.cc:503] Master 127.23.173.126:38979 was elected leader, sending a full tablet report...
I20260927 22:17:08.890986 3610 consensus_queue.cc:260] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.891987 3609 raft_consensus.cc:374] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:17:08.892156 3609 raft_consensus.cc:729] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7209bc363c654b78825d4d84c21f6ba9, State: Initialized, Role: FOLLOWER
I20260927 22:17:08.892197 3608 ts_tablet_manager.cc:1428] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260927 22:17:08.892856 3609 consensus_queue.cc:260] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:08.895557 3610 ts_tablet_manager.cc:1428] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Time spent starting tablet: real 0.019s user 0.011s sys 0.006s
I20260927 22:17:08.896780 3455 heartbeater.cc:503] Master 127.23.173.126:38979 was elected leader, sending a full tablet report...
I20260927 22:17:08.897439 3609 ts_tablet_manager.cc:1428] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Time spent starting tablet: real 0.022s user 0.010s sys 0.005s
W20260927 22:17:08.901382 3456 tablet.cc:2369] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:17:08.987327 3313 tablet.cc:2369] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:17:08.995802 3590 tablet.cc:2369] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:17:09.039824 3615 raft_consensus.cc:482] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:17:09.040159 3615 raft_consensus.cc:504] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:09.041874 3615 leader_election.cc:288] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6049ddd7c48c461b93b0953b622df0dd (127.23.173.67:42519), 7209bc363c654b78825d4d84c21f6ba9 (127.23.173.66:39553)
I20260927 22:17:09.049687 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.042167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:39855 (local address 127.23.173.67:42519)
0927 22:17:09.042402 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.042427 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.043091 (+ 664us) server_negotiation.cc:407] Connection header received
0927 22:17:09.043329 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.043333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.043392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.043494 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:17:09.044035 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.044738 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.047323 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.048584 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.048755 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.048828 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.049302 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.049428 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:17:09.049687 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.042448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:60653 (local address 127.23.173.66:39553)
0927 22:17:09.042682 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.042691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.043342 (+ 651us) server_negotiation.cc:407] Connection header received
0927 22:17:09.043553 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.043557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.043659 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.043753 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:17:09.044339 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.045013 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.047421 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.048606 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.048760 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.048828 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.049302 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.049428 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:17:09.049926 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.042361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:39553 (local address 127.23.173.65:60653)
0927 22:17:09.043225 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:09.043239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:09.043394 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:09.043770 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:09.043774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:09.043802 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:17:09.044171 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:09.044180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.045235 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.045240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:09.047279 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:09.047289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.048880 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.048885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:09.049085 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.049173 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 22:17:09.049263 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:17:09.049427 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":54,"thread_start_us":506,"threads_started":1}
I20260927 22:17:09.050051 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.042109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:42519 (local address 127.23.173.65:39855)
0927 22:17:09.042965 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:09.042983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:09.043173 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:09.043523 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:09.043527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:09.043583 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:17:09.043887 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:09.043896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.044865 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.044872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:09.047163 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:09.047182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.048870 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.048877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:09.049085 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.049167 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 22:17:09.049263 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:17:09.049427 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":86,"thread_start_us":552,"threads_started":1}
I20260927 22:17:09.050944 3410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476ba128e57d4639a9790613c38f9610" candidate_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7209bc363c654b78825d4d84c21f6ba9" is_pre_election: true
I20260927 22:17:09.051043 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476ba128e57d4639a9790613c38f9610" candidate_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6049ddd7c48c461b93b0953b622df0dd" is_pre_election: true
I20260927 22:17:09.051554 3410 raft_consensus.cc:2454] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76faf2676b1645fb8d1d1b6859dabb0f in term 0.
I20260927 22:17:09.051561 3544 raft_consensus.cc:2454] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76faf2676b1645fb8d1d1b6859dabb0f in term 0.
I20260927 22:17:09.052197 3202 leader_election.cc:302] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6049ddd7c48c461b93b0953b622df0dd, 7209bc363c654b78825d4d84c21f6ba9, 76faf2676b1645fb8d1d1b6859dabb0f; no voters:
I20260927 22:17:09.052620 3615 raft_consensus.cc:2781] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:17:09.052762 3615 raft_consensus.cc:482] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:17:09.052873 3615 raft_consensus.cc:3037] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:17:09.055150 3615 raft_consensus.cc:504] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:09.055769 3615 leader_election.cc:288] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [CANDIDATE]: Term 1 election: Requested vote from peers 6049ddd7c48c461b93b0953b622df0dd (127.23.173.67:42519), 7209bc363c654b78825d4d84c21f6ba9 (127.23.173.66:39553)
I20260927 22:17:09.056195 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476ba128e57d4639a9790613c38f9610" candidate_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6049ddd7c48c461b93b0953b622df0dd"
I20260927 22:17:09.056255 3410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476ba128e57d4639a9790613c38f9610" candidate_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7209bc363c654b78825d4d84c21f6ba9"
I20260927 22:17:09.056380 3544 raft_consensus.cc:3037] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:17:09.056509 3410 raft_consensus.cc:3037] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:17:09.058696 3544 raft_consensus.cc:2454] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76faf2676b1645fb8d1d1b6859dabb0f in term 1.
I20260927 22:17:09.058754 3410 raft_consensus.cc:2454] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76faf2676b1645fb8d1d1b6859dabb0f in term 1.
I20260927 22:17:09.059124 3203 leader_election.cc:302] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6049ddd7c48c461b93b0953b622df0dd, 76faf2676b1645fb8d1d1b6859dabb0f; no voters:
I20260927 22:17:09.059410 3615 raft_consensus.cc:2781] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:17:09.059820 3615 raft_consensus.cc:686] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 1 LEADER]: Becoming Leader. State: Replica: 76faf2676b1645fb8d1d1b6859dabb0f, State: Running, Role: LEADER
I20260927 22:17:09.060297 3615 consensus_queue.cc:237] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } }
I20260927 22:17:09.065024 3097 catalog_manager.cc:5495] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 76faf2676b1645fb8d1d1b6859dabb0f (127.23.173.65). New cstate: current_term: 1 leader_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76faf2676b1645fb8d1d1b6859dabb0f" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 39005 } health_report { overall_health: HEALTHY } } }
I20260927 22:17:09.135744 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.130190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.23.173.65:39005)
0927 22:17:09.130407 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.130414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.130433 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:17:09.130680 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.130684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.130788 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.130953 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:17:09.131306 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.132004 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.133447 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.133847 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.133934 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.134614 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:17:09.135116 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:17:09.135232 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.135364 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.135457 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:17:09.144913 3544 raft_consensus.cc:1261] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 1 FOLLOWER]: Refusing update from remote peer 76faf2676b1645fb8d1d1b6859dabb0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:17:09.146021 3410 raft_consensus.cc:1261] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 1 FOLLOWER]: Refusing update from remote peer 76faf2676b1645fb8d1d1b6859dabb0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:17:09.145929 3619 consensus_queue.cc:1035] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6049ddd7c48c461b93b0953b622df0dd" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:17:09.146795 3619 consensus_queue.cc:1035] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7209bc363c654b78825d4d84c21f6ba9" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:17:09.161793 3624 mvcc.cc:205] Tried to move back new op lower bound from 7334082269767221248 to 7334082269434560512. Current Snapshot: MvccSnapshot[applied={T|T < 7334082269767221248}]
I20260927 22:17:09.176579 3099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:17:09.176889 3099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:17:09.180941 3099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ad2a0c1007f46cfa3e5e315f771736f: Sending DeleteTablet for 3 replicas of tablet 476ba128e57d4639a9790613c38f9610
I20260927 22:17:09.182454 3390 tablet_service.cc:1514] Processing DeleteTablet for tablet 476ba128e57d4639a9790613c38f9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:17:09 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:56638
I20260927 22:17:09.182688 3247 tablet_service.cc:1514] Processing DeleteTablet for tablet 476ba128e57d4639a9790613c38f9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:17:09 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:57198
I20260927 22:17:09.182729 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 476ba128e57d4639a9790613c38f9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:17:09 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:41066
I20260927 22:17:09.188117 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 3179
I20260927 22:17:09.202579 3633 tablet_replica.cc:330] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: stopping tablet replica
I20260927 22:17:09.203070 3633 raft_consensus.cc:2229] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:17:09.203694 3633 raft_consensus.cc:2258] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:17:09.206167 3633 ts_tablet_manager.cc:1905] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:17:09.214592 3633 ts_tablet_manager.cc:1918] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:17:09.214864 3633 log.cc:1196] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/476ba128e57d4639a9790613c38f9610
I20260927 22:17:09.215314 3634 tablet_replica.cc:330] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: stopping tablet replica
I20260927 22:17:09.215597 3633 ts_tablet_manager.cc:1939] T 476ba128e57d4639a9790613c38f9610 P 76faf2676b1645fb8d1d1b6859dabb0f: Deleting consensus metadata
I20260927 22:17:09.215948 3634 raft_consensus.cc:2229] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:17:09.216614 3634 raft_consensus.cc:2258] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:17:09.218040 3086 catalog_manager.cc:4841] TS 76faf2676b1645fb8d1d1b6859dabb0f (127.23.173.65:39005): tablet 476ba128e57d4639a9790613c38f9610 (table test-table [id=f767e8f1247e4ba99c4510c938e332f6]) successfully deleted
I20260927 22:17:09.221814 3632 tablet_replica.cc:330] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: stopping tablet replica
I20260927 22:17:09.222329 3632 raft_consensus.cc:2229] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:17:09.222913 3632 raft_consensus.cc:2258] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:17:09.223413 3634 ts_tablet_manager.cc:1905] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:17:09.224999 3632 ts_tablet_manager.cc:1905] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:17:09 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:17:09.230355 3634 ts_tablet_manager.cc:1918] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:17:09.230502 3634 log.cc:1196] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/476ba128e57d4639a9790613c38f9610
I20260927 22:17:09.230958 3634 ts_tablet_manager.cc:1939] T 476ba128e57d4639a9790613c38f9610 P 6049ddd7c48c461b93b0953b622df0dd: Deleting consensus metadata
I20260927 22:17:09.232637 3086 catalog_manager.cc:4841] TS 6049ddd7c48c461b93b0953b622df0dd (127.23.173.67:42519): tablet 476ba128e57d4639a9790613c38f9610 (table test-table [id=f767e8f1247e4ba99c4510c938e332f6]) successfully deleted
I20260927 22:17:09.236958 3632 ts_tablet_manager.cc:1918] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:17:09.237123 3632 log.cc:1196] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/476ba128e57d4639a9790613c38f9610
I20260927 22:17:09.237586 3632 ts_tablet_manager.cc:1939] T 476ba128e57d4639a9790613c38f9610 P 7209bc363c654b78825d4d84c21f6ba9: Deleting consensus metadata
I20260927 22:17:09.238617 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.189971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.23.173.65:39005)
0927 22:17:09.190200 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.190213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.211616 (+ 21403us) server_negotiation.cc:407] Connection header received
0927 22:17:09.212231 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.212240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.212375 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.212476 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:09.217106 (+ 4630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.218600 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.221199 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.222095 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.222247 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.228721 (+ 6474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:09.228738 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:09.228756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:09.228806 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:09.234094 (+ 5288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.235198 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.235209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.235213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.235420 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.235931 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.235949 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.235953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.236450 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:09.236755 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.237623 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.238076 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:17:09.239279 3085 catalog_manager.cc:4841] TS 7209bc363c654b78825d4d84c21f6ba9 (127.23.173.66:39553): tablet 476ba128e57d4639a9790613c38f9610 (table test-table [id=f767e8f1247e4ba99c4510c938e332f6]) successfully deleted
I20260927 22:17:09.240415 3307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:09.343124 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 3179
I20260927 22:17:09.362157 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 3318
Sep 27 22:17:09 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:17:09.376849 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.363055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.23.173.66:39553)
0927 22:17:09.363296 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.363307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.363345 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:17:09.363476 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.363480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.363553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.363641 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:09.364937 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.365693 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.367100 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.367473 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.367569 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.371584 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:09.371607 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:09.371623 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:09.371657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:09.374402 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.375006 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.375013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.375016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.375145 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.375474 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.375477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.375479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.375830 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:09.376039 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.376323 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.376478 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:17:09.377559 3450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:09.467351 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 3318
I20260927 22:17:09.484162 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 3461
Sep 27 22:17:09 dist-test-slave-3wc3 krb5kdc[2623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:17:09.498867 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:09.484938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.23.173.67:42519)
0927 22:17:09.485134 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:09.485139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:09.485158 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:17:09.485278 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:09.485282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:09.485343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:09.485424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:09.486768 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.487530 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.488898 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:09.489272 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:09.489361 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:09.493603 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:09.493617 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:09.493633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:09.493673 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:09.496467 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.497049 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.497065 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.497067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.497196 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:09.497539 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:09.497542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:09.497544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:09.497881 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:09.498167 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:09.498365 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:17:09.498518 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:17:09.499518 3584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:09.587489 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 3461
I20260927 22:17:09.603410 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 3067
I20260927 22:17:09.604584 3129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:09.717249 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 3067
I20260927 22:17:09.741118 24245 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:17:10.076406 24245 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:17:10.076609 24245 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:17:10.423179 24245 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:17:10.423 UTC [2653] LOG: received smart shutdown request
2026-09-27 22:17:10.427 UTC [2653] LOG: background worker "logical replication launcher" (PID 2662) exited with exit code 1
2026-09-27 22:17:10.427 UTC [2657] LOG: shutting down
2026-09-27 22:17:10.612 UTC [2653] LOG: database system is shut down
2026-09-27T22:17:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49449 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:17:10 dist-test-slave-3wc3 krb5kdc[3648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:17:10 dist-test-slave-3wc3 krb5kdc[3648](info): set up 2 sockets
Sep 27 22:17:10 dist-test-slave-3wc3 krb5kdc[3648](info): commencing operation
krb5kdc: starting...
W20260927 22:17:11.862440 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:17:12 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:17:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangeradmin_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangeradmin_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangerlookup_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangerlookup_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
I20260927 22:17:12.189069 24245 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskO6uUAX/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/postgres -l logfile start
2026-09-27 22:17:14.695 UTC [3678] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:17:14.695 UTC [3678] LOG: listening on IPv4 address "127.23.173.84", port 39707
2026-09-27 22:17:14.698 UTC [3678] LOG: listening on Unix socket "/tmp/.s.PGSQL.39707"
2026-09-27 22:17:14.721 UTC [3681] LOG: database system was shut down at 2026-09-27 22:17:13 UTC
2026-09-27 22:17:14.725 UTC [3678] LOG: database system is ready to accept connections
I20260927 22:17:15.693946 24245 mini_postgres.cc:96] Postgres bound to 39707
2026-09-27 22:17:15.715 UTC [3689] FATAL: database "slave" does not exist
127.23.173.84:39707 - accepting connections
I20260927 22:17:15.716348 24245 mini_ranger.cc:161] Starting Ranger...
I20260927 22:17:15.742369 24245 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:17:16.220209 24245 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:17:16.220302 24245 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin
2026-09-27 22:17:16,528 [I] DB FLAVOR :POSTGRES
2026-09-27 22:17:16,529 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:17:16,529 [I] Checking connection..
2026-09-27 22:17:16,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:17:16,776 [I] Checking connection passed.
2026-09-27 22:17:16,777 [I] --------- Verifying version history table ---------
2026-09-27 22:17:16,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:17:17,009 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:17:17,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:17:17,268 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:17:17,268 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:17:17,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:17:17.502 UTC [3753] WARNING: there is no transaction in progress
2026-09-27 22:17:17,509 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:17:17,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:17:17,757 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:17:17,758 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:17:17,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:17:18,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:17:18,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3wc3.c.gcp-upstream.internal','N') ;"
2026-09-27 22:17:18,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:17:18,756 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:17:18,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:17:19,027 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:17:19,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:17:19,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:17:19,521 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:17:19,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:17:20.428 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.444 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.460 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.475 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.490 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.507 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.570 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.576 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.590 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.600 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.611 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.625 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.634 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.640 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.647 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:20.653 UTC [3861] WARNING: there is no transaction in progress
2026-09-27 22:17:21,265 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:17:21,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:17:21,492 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:17:21,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:17:21,733 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:17:21,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:17:21,994 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:17:21,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:17:22,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:17:22,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3wc3.c.gcp-upstream.internal';"
2026-09-27 22:17:22,735 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:17:22,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:17:22,981 [I] DB_PATCHES have already been applied
I20260927 22:17:22.987843 24245 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskO6uUAX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:17:22.987922 24245 mini_ranger.cc:193] Using host: 127.23.173.84
I20260927 22:17:23.002449 24245 mini_ranger.cc:227] Ranger admin URL: http://127.23.173.84:44223
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:17:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:17:24.127440 24245 mini_ranger.cc:160] Time spent starting Ranger: real 8.411s user 0.002s sys 0.036s
Sep 27, 2026 10:17:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:17:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 10:17:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:17:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:17:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:17:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:17:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:17:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:17:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:17:44.462383 24245 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.173.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.173.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangerkms_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.173.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/rangerkms_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.173.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.173.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.173.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/HTTP_127.23.173.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:17:44 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547464, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:17:44.624680 24245 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:17:44.647902 24245 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:17:45.509534 24245 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:17:45.509666 24245 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms
I20260927 22:17:45.509732 24245 mini_ranger_kms.cc:227] Using postgres at 127.23.173.84:39707
2026-09-27 22:17:45,864 [I] DB FLAVOR :POSTGRES
2026-09-27 22:17:45,864 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:17:45,864 [I] Checking connection
2026-09-27 22:17:45,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:17:46,097 [I] connection success
2026-09-27 22:17:46,098 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:17:46,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:17:46,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:17:46,343 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:17:46,343 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:17:46,343 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:17:46,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.173.84:39707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:17:46,622 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:17:47.099617 24245 mini_ranger_kms.cc:310] Created kms service
I20260927 22:17:47.322531 24245 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:17:47.367132 24245 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:17:47 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547464, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:47.781832 24245 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:17:47.782166 24245 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms:/tmp/dist-test-taskO6uUAX/build/asan/bin/postgresql.jar:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskO6uUAX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:17:47.782212 24245 mini_ranger_kms.cc:240] Using host: 127.23.173.84
I20260927 22:17:47.801316 24245 mini_ranger_kms.cc:276] Ranger KMS PID: 4041
I20260927 22:17:47.801468 24245 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.173.84:56263
Sep 27, 2026 10:17:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:17:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:17:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:17:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:17:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:17:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:17:50 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547470, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:50.174803 24245 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.550s user 0.087s sys 0.280s
I20260927 22:17:50.175060 24245 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:17:50.175230 24245 mini_ranger_kms.cc:401] 127.23.173.84:56263/kms/v1/keys
Sep 27 22:17:50 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547470, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
Sep 27 22:17:50 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547470, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.173.84@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
Sep 27 22:17:53 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:17:53.161386 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35911
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:35911
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:17:53.496323 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:53.496616 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:53.496783 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:53.503772 4092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:17:53.503881 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:53.503926 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:53.504005 4092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:17:53.504113 4092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:17:53.513769 4092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.173.84:56263/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:35911
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:53.514989 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:53.516315 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:53.528093 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:53.528156 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:53.528060 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:53.528368 4092 server_base.cc:1034] running on GCE node
I20260927 22:17:53.529341 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:53.530680 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:53.531899 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790547473531864 us; error 44 us; skew 500 ppm
I20260927 22:17:53.532222 4092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:53 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:53.535131 4092 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:17:53.536522 4092 webserver.cc:460] Webserver started at http://127.23.173.126:35625/ using document root <none> and password file <none>
I20260927 22:17:53.537127 4092 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:53.537252 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:53.537549 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:53 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:53.584612 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "5213b9ec87f8438b92c35d5adbdb814d"
format_stamp: "Formatted at 2026-09-27 22:17:53 on dist-test-slave-3wc3"
server_key: "a061fb96813c2febfb63b791508665ec"
server_key_iv: "c336d68b9b242ac538d1fc0029961aa6"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:53.585440 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "5213b9ec87f8438b92c35d5adbdb814d"
format_stamp: "Formatted at 2026-09-27 22:17:53 on dist-test-slave-3wc3"
server_key: "a061fb96813c2febfb63b791508665ec"
server_key_iv: "c336d68b9b242ac538d1fc0029961aa6"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:53.590924 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:17:53.606022 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:53.607486 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:17:53.607626 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "5213b9ec87f8438b92c35d5adbdb814d"
format_stamp: "Formatted at 2026-09-27 22:17:53 on dist-test-slave-3wc3"
server_key: "a061fb96813c2febfb63b791508665ec"
server_key_iv: "c336d68b9b242ac538d1fc0029961aa6"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:53.607808 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:53.621803 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:53.625285 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:53.625524 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:53.643550 4092 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:35911
I20260927 22:17:53.643571 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:35911 every 8 connection(s)
I20260927 22:17:53.644963 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:17:53.648679 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:17:53.654850 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4092
I20260927 22:17:53.655061 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:17:53.655721 24245 external_mini_cluster.cc:1428] Setting key 8a4bd1bcab1605c1d1499dbb7aac4fc6
I20260927 22:17:53.659101 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Bootstrap starting.
I20260927 22:17:53.662894 4159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Neither blocks nor log segments found. Creating new log.
I20260927 22:17:53.664129 4159 log.cc:824] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:53.666738 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: No bootstrap required, opened a new log
Sep 27 22:17:53 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:53.674201 4159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:53.674540 4159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:17:53.674639 4159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5213b9ec87f8438b92c35d5adbdb814d, State: Initialized, Role: FOLLOWER
I20260927 22:17:53.675275 4159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:53.675551 4159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:17:53.675688 4159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:17:53.675876 4159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:17:53.678452 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:53.657652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.23.173.126:35911)
0927 22:17:53.658503 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:53.658521 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:53.658568 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:17:53.661014 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:53.661042 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:53.661518 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:53.661930 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:53.663247 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:53.665134 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:53.666536 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:53.667232 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:53.667442 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:53.671130 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:53.671138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:53.671151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:53.671190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:53.674117 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:53.674859 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:53.674867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:53.674873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:53.675051 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:53.675412 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:53.675417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:53.675420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:53.676155 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:53.676475 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:53.676806 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:17:53.677210 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":344,"threads_started":1}
I20260927 22:17:53.679057 4159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:53.679603 4159 leader_election.cc:302] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5213b9ec87f8438b92c35d5adbdb814d; no voters:
I20260927 22:17:53.680210 4159 leader_election.cc:288] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:17:53.680357 4164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:17:53.680995 4164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 LEADER]: Becoming Leader. State: Replica: 5213b9ec87f8438b92c35d5adbdb814d, State: Running, Role: LEADER
I20260927 22:17:53.681582 4164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:53.683369 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5213b9ec87f8438b92c35d5adbdb814d. Latest consensus state: current_term: 1 leader_uuid: "5213b9ec87f8438b92c35d5adbdb814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } } }
I20260927 22:17:53.683781 4159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:17:53.683871 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:53.684296 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5213b9ec87f8438b92c35d5adbdb814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } } }
I20260927 22:17:53.684649 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:53.687474 4168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:17:53.692600 4168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:17:53.696305 4159 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:17:53.709693 4168 catalog_manager.cc:1347] Generated new cluster ID: 4fa768bae899430caf537f8f8d9bff9a
I20260927 22:17:53.709841 4168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:17:53.725572 4168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:17:53.727268 4168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:17:53.733197 4168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Generated new TSK 0
I20260927 22:17:53.734118 4168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:17:54.577311 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:54 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:17:54.984876 4204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:54.985263 4204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:54.985455 4204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:54.996948 4204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:54.997138 4204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:54.997368 4204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:17:55.013036 4204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:55.015036 4204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:55.016837 4204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:55.033198 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:55.034587 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:55.041832 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:55.042954 4204 server_base.cc:1034] running on GCE node
I20260927 22:17:55.043718 4204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:55.044970 4204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:55.046252 4204 hybrid_clock.cc:648] HybridClock initialized: now 1790547475046188 us; error 46 us; skew 500 ppm
I20260927 22:17:55.046630 4204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:55.050968 4204 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:17:55.053026 4204 webserver.cc:460] Webserver started at http://127.23.173.65:42447/ using document root <none> and password file <none>
I20260927 22:17:55.053936 4204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:55.054102 4204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:55.054435 4204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:55.073242 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "2393e79232c24b2fb0078dfef1468b09"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "3dd0a5b65778cf86531ef901f6203abb"
server_key_iv: "41c33dd908377647f71b6eeb007769f5"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:55.074414 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "2393e79232c24b2fb0078dfef1468b09"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "3dd0a5b65778cf86531ef901f6203abb"
server_key_iv: "41c33dd908377647f71b6eeb007769f5"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:55.081204 4204 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:17:55.104635 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:55.106344 4204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:17:55.106585 4204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "2393e79232c24b2fb0078dfef1468b09"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "3dd0a5b65778cf86531ef901f6203abb"
server_key_iv: "41c33dd908377647f71b6eeb007769f5"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:55.106953 4204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:55.128166 4204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:55.133872 4204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:55.134274 4204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:55.136801 4204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:55.138513 4204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:55.138628 4204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:55.138777 4204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:55.138854 4204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:55.243309 4204 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:42723
I20260927 22:17:55.243372 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:42723 every 8 connection(s)
I20260927 22:17:55.244786 4204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:17:55.248883 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4204
I20260927 22:17:55.249102 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:17:55.249908 24245 external_mini_cluster.cc:1428] Setting key 17fa8f9c7d52e5ac7934d32bdc0a1091
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
I20260927 22:17:55.306810 4339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:55.249826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:37981 (local address 127.23.173.126:35911)
0927 22:17:55.268115 (+ 18289us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:55.268122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:55.282044 (+ 13922us) server_negotiation.cc:407] Connection header received
0927 22:17:55.282145 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:55.282150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:55.282251 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:55.282378 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:55.284645 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:55.285846 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:55.288542 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:55.289372 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:55.289483 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:55.294730 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:55.294740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:55.294748 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:55.294796 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:55.298696 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:55.302638 (+ 3942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:55.302646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:55.302653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:55.302851 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:55.303488 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:55.303495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:55.303500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:55.303905 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:55.304035 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:55.306175 (+ 2140us) server_negotiation.cc:299] Negotiation successful
0927 22:17:55.306460 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18139,"thread_start_us":18288,"threads_started":1}
I20260927 22:17:55.309280 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:55.250885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.65:37981)
0927 22:17:55.274559 (+ 23674us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:55.274615 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:55.280364 (+ 5749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:55.282864 (+ 2500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:55.282878 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:55.283533 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:55.284351 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:55.284378 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:55.286092 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:55.286097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:55.288384 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:55.288400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:55.289834 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:55.289843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:55.290195 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:55.290913 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:55.291012 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:55.294464 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:55.301505 (+ 7041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:55.301517 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:55.301540 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:55.302409 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:55.303030 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:55.303038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:55.303046 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:55.303342 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:55.304154 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:55.304162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:55.304672 (+ 510us) client_negotiation.cc:769] Sending connection context
0927 22:17:55.307068 (+ 2396us) client_negotiation.cc:240] Negotiation successful
0927 22:17:55.307806 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23260,"thread_start_us":24632,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
I20260927 22:17:55.311977 4337 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:55.312327 4337 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:55.313262 4337 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:55.321893 4124 ts_manager.cc:194] Registered new tserver with Master: 2393e79232c24b2fb0078dfef1468b09 (127.23.173.65:42723)
I20260927 22:17:55.324082 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:37981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:17:55.353271 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547474, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
W20260927 22:17:55.815115 4343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:55.815642 4343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:55.815845 4343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:55.827536 4343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:55.827737 4343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:55.827982 4343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:17:55.843318 4343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:55.845291 4343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:55.847021 4343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:55.891685 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:55.898667 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:55.899420 4354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:55.900549 4343 server_base.cc:1034] running on GCE node
I20260927 22:17:55.901538 4343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:55.910243 4343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:55.918763 4343 hybrid_clock.cc:648] HybridClock initialized: now 1790547475918626 us; error 131 us; skew 500 ppm
I20260927 22:17:55.919288 4343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:55.923990 4343 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:17:55.948010 4343 webserver.cc:460] Webserver started at http://127.23.173.66:38183/ using document root <none> and password file <none>
I20260927 22:17:55.948957 4343 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:55.949188 4343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:55.949635 4343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:55 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:55.992070 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "ec32e167aea04eb0bb0a3201df86241a"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "0782adde4e8d4f492b86aedb45acd17a"
server_key_iv: "adf475d0fc4158fe60da73a198ca45e0"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:55.993564 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "ec32e167aea04eb0bb0a3201df86241a"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "0782adde4e8d4f492b86aedb45acd17a"
server_key_iv: "adf475d0fc4158fe60da73a198ca45e0"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:56.007436 4343 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260927 22:17:56.046571 4367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.048316 4343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 22:17:56.048522 4343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "ec32e167aea04eb0bb0a3201df86241a"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "0782adde4e8d4f492b86aedb45acd17a"
server_key_iv: "adf475d0fc4158fe60da73a198ca45e0"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:56.048761 4343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:56.131513 4343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:56.136912 4343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:56.137307 4343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:56.150471 4343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:56.158672 4343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:56.158866 4343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.159065 4343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:56.159198 4343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.290973 4343 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:40235
I20260927 22:17:56.291014 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:40235 every 8 connection(s)
I20260927 22:17:56.293017 4343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:17:56.302906 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4343
I20260927 22:17:56.303063 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:17:56.303577 24245 external_mini_cluster.cc:1428] Setting key 2da887f464a7656301ac84f16f86fb50
Sep 27 22:17:56 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547475, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:56.323043 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:56.297178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:44539 (local address 127.23.173.126:35911)
0927 22:17:56.302377 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:56.302383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:56.302407 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:17:56.304416 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:56.304422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:56.304507 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:56.304614 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:56.306748 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.307791 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.310137 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.310818 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.310912 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:56.315454 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:56.315461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:56.315465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:56.315509 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:56.319090 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.320284 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.320290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.320293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.320440 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.320944 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.320949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.320952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.321301 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:56.321428 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:56.322406 (+ 978us) server_negotiation.cc:299] Negotiation successful
0927 22:17:56.322651 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5068,"thread_start_us":4978,"threads_started":1}
I20260927 22:17:56.325619 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:56.296600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.66:44539)
0927 22:17:56.301021 (+ 4421us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:56.301072 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:56.303932 (+ 2860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:56.305127 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:56.305140 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:56.305750 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:56.306566 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:56.306590 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.307995 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.308000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:56.309949 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:56.309972 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.311238 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.311245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:56.311518 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:56.312506 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:56.312535 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:56.315197 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:56.319530 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:56.319540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:56.319557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:56.320117 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:56.320581 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:56.320586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:56.320589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:56.320822 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:56.321517 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:56.321525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:56.321938 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 22:17:56.323610 (+ 1672us) client_negotiation.cc:240] Negotiation successful
0927 22:17:56.324238 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4095,"thread_start_us":3842,"threads_started":1}
I20260927 22:17:56.327616 4480 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:56.327872 4337 heartbeater.cc:503] Master 127.23.173.126:35911 was elected leader, sending a full tablet report...
I20260927 22:17:56.327955 4480 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:56.328890 4480 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:56.330813 4124 ts_manager.cc:194] Registered new tserver with Master: ec32e167aea04eb0bb0a3201df86241a (127.23.173.66:40235)
I20260927 22:17:56.331835 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:44539
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:17:56.371454 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:17:56.738059 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:56.738344 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:56.738533 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:56.745754 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:56.745890 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:56.746088 4486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:17:56.755806 4486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:56.757112 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:56.758406 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:56.771008 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:56.771090 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:56.771059 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:56.771771 4486 server_base.cc:1034] running on GCE node
I20260927 22:17:56.772341 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:56.773248 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:56.774467 4486 hybrid_clock.cc:648] HybridClock initialized: now 1790547476774416 us; error 63 us; skew 500 ppm
I20260927 22:17:56.774768 4486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:56 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547476, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:56.777667 4486 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:17:56.779124 4486 webserver.cc:460] Webserver started at http://127.23.173.67:38217/ using document root <none> and password file <none>
I20260927 22:17:56.779795 4486 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:56.779933 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:17:56.780324 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:17:56 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547476, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:56.795403 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "52600e480ea64333b9eb18e647e3725e"
format_stamp: "Formatted at 2026-09-27 22:17:56 on dist-test-slave-3wc3"
server_key: "c4a522575abc2d0d12714869b6dff369"
server_key_iv: "169b4a3feb9f90a51b7758244d1c71e9"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:56.796262 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "52600e480ea64333b9eb18e647e3725e"
format_stamp: "Formatted at 2026-09-27 22:17:56 on dist-test-slave-3wc3"
server_key: "c4a522575abc2d0d12714869b6dff369"
server_key_iv: "169b4a3feb9f90a51b7758244d1c71e9"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:56.802376 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 22:17:56.818395 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.819885 4486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:17:56.820025 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "52600e480ea64333b9eb18e647e3725e"
format_stamp: "Formatted at 2026-09-27 22:17:56 on dist-test-slave-3wc3"
server_key: "c4a522575abc2d0d12714869b6dff369"
server_key_iv: "169b4a3feb9f90a51b7758244d1c71e9"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:56.820271 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:56.835932 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:56.839762 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:56.840049 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:56.840977 4486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:56.842357 4486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:56.842427 4486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.842587 4486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:56.842653 4486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:56.873077 4486 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:40559
I20260927 22:17:56.873111 4613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:40559 every 8 connection(s)
I20260927 22:17:56.874401 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:17:56.878674 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4486
I20260927 22:17:56.878834 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:17:56.879277 24245 external_mini_cluster.cc:1428] Setting key ee8f087d70960727385b62439cf5d943
Sep 27 22:17:56 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547476, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:56.893563 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:56.876884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:51931 (local address 127.23.173.126:35911)
0927 22:17:56.877381 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:56.877386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:56.878249 (+ 863us) server_negotiation.cc:407] Connection header received
0927 22:17:56.880152 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:56.880159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:56.880284 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:56.880400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:56.881742 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.882580 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.884167 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.884622 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.884755 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:56.887983 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:56.887989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:56.887994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:56.888031 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:56.890413 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.891105 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.891110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.891118 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.891239 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.891663 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.891668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.891671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.891924 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:56.892084 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:56.892942 (+ 858us) server_negotiation.cc:299] Negotiation successful
0927 22:17:56.893163 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260927 22:17:56.894716 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:56.877351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.67:51931)
0927 22:17:56.878088 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:56.878125 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:56.879895 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:56.880634 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:56.880642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:56.881030 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:56.881581 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:56.881597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.882740 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.882745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:56.884017 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:56.884035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.884843 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.884850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:56.885086 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:56.885662 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:56.885683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:56.887790 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:56.890538 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:56.890546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:56.890561 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:56.890983 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:56.891344 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:56.891349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:56.891353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:56.891546 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:56.892074 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:56.892081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:56.892546 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 22:17:56.892863 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 22:17:56.893367 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
I20260927 22:17:56.896076 4614 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:56.896389 4614 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:56.897045 4614 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:56.898453 4124 ts_manager.cc:194] Registered new tserver with Master: 52600e480ea64333b9eb18e647e3725e (127.23.173.67:40559)
I20260927 22:17:56.899257 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:51931
I20260927 22:17:56.906371 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:17:56.908084 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4204
Sep 27 22:17:56 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:17:56.924530 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:56.908919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.23.173.65:42723)
0927 22:17:56.909536 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:56.909547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:56.909578 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:17:56.909784 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:56.909791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:56.909986 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:56.910147 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:56.911641 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.912741 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.914287 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:56.914817 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:56.914949 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:56.919104 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:56.919115 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:56.919127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:56.919159 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:56.921742 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.922456 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.922463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.922465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.922598 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:56.922988 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:56.922992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:56.922994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:56.923335 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:56.923636 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:56.923930 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 22:17:56.924190 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
I20260927 22:17:56.925707 4332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:57.007632 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4204
I20260927 22:17:57.023792 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4343
Sep 27 22:17:57 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:17:57.039392 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:57.024641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.23.173.66:40235)
0927 22:17:57.025230 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:57.025237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:57.025263 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:17:57.025344 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:57.025350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:57.025556 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:57.025688 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:57.026998 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:57.027972 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:57.029206 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:57.029741 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:57.029836 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:57.034030 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:57.034041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:57.034055 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:57.034085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:57.036587 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:57.037283 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:57.037290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:57.037292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:57.037426 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:57.037776 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:57.037781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:57.037785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:57.038165 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:57.038479 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:57.038759 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 22:17:57.039022 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":324,"threads_started":1}
I20260927 22:17:57.040591 4475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:57.121897 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4343
I20260927 22:17:57.137117 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4486
Sep 27 22:17:57 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:17:57.153281 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:57.138024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.23.173.67:40559)
0927 22:17:57.138622 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:57.138629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:57.138658 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:17:57.138716 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:57.138722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:57.138941 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:57.139106 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:57.140531 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:57.141545 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:57.142914 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:57.143473 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:57.143612 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:57.147752 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:57.147763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:57.147777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:57.147812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:57.151005 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:57.151605 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:57.151611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:57.151614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:57.151733 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:57.152043 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:57.152046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:57.152048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:57.152364 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:57.152594 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:57.152813 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 22:17:57.153000 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
I20260927 22:17:57.154592 4609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:57.236491 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4486
I20260927 22:17:57.251911 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4092
I20260927 22:17:57.253193 4154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:17:57.360406 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4092
I20260927 22:17:57.381629 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35911
--webserver_interface=127.23.173.126
--webserver_port=35625
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:35911
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:17:57.703575 4626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:57.703905 4626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:57.704059 4626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:57.711144 4626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:17:57.711290 4626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:57.711396 4626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:57.711491 4626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:17:57.711568 4626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:17:57.721349 4626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:35911
--ranger_config_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:35911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=35625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:57.722749 4626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:57.724046 4626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:57.735553 4631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:57.735616 4632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:57.735608 4634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:57.736364 4626 server_base.cc:1034] running on GCE node
I20260927 22:17:57.737025 4626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:57.738332 4626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:57.739528 4626 hybrid_clock.cc:648] HybridClock initialized: now 1790547477739490 us; error 25 us; skew 500 ppm
I20260927 22:17:57.739814 4626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:57 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:57.742774 4626 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:17:57.744167 4626 webserver.cc:460] Webserver started at http://127.23.173.126:35625/ using document root <none> and password file <none>
I20260927 22:17:57.744777 4626 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:57.744908 4626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:17:57 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:57.799475 4626 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:17:57.802981 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:57.804353 4626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:17:57.804551 4626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "5213b9ec87f8438b92c35d5adbdb814d"
format_stamp: "Formatted at 2026-09-27 22:17:53 on dist-test-slave-3wc3"
server_key: "a061fb96813c2febfb63b791508665ec"
server_key_iv: "c336d68b9b242ac538d1fc0029961aa6"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:57.805513 4626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:57.841279 4626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:57.845400 4626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:57.845686 4626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:57.864234 4626 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:35911
I20260927 22:17:57.864243 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:35911 every 8 connection(s)
I20260927 22:17:57.865734 4626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:17:57.872722 4694 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:17:57.872890 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4626
I20260927 22:17:57.873764 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:42723
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:35911
--webserver_port=42447
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:17:57.876200 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Bootstrap starting.
I20260927 22:17:57.894791 4694 log.cc:824] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:17:57.903482 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:17:57.903985 4694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Bootstrap complete.
I20260927 22:17:57.912927 4694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:57.913774 4694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5213b9ec87f8438b92c35d5adbdb814d, State: Initialized, Role: FOLLOWER
I20260927 22:17:57.914376 4694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:57.914602 4694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:17:57.914717 4694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:17:57.914901 4694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:17:57.918617 4694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:57.919107 4694 leader_election.cc:302] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5213b9ec87f8438b92c35d5adbdb814d; no voters:
I20260927 22:17:57.919749 4694 leader_election.cc:288] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:17:57.919893 4698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:17:57.921196 4698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [term 2 LEADER]: Becoming Leader. State: Replica: 5213b9ec87f8438b92c35d5adbdb814d, State: Running, Role: LEADER
I20260927 22:17:57.921731 4698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } }
I20260927 22:17:57.921986 4694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:17:57.924345 4699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5213b9ec87f8438b92c35d5adbdb814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } } }
I20260927 22:17:57.924767 4699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:57.929541 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5213b9ec87f8438b92c35d5adbdb814d. Latest consensus state: current_term: 2 leader_uuid: "5213b9ec87f8438b92c35d5adbdb814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5213b9ec87f8438b92c35d5adbdb814d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 35911 } } }
I20260927 22:17:57.930495 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d [sys.catalog]: This master's current role is: LEADER
I20260927 22:17:57.932674 4704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:17:57.938481 4704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:17:57.940083 4704 catalog_manager.cc:1259] Loaded cluster ID: 4fa768bae899430caf537f8f8d9bff9a
I20260927 22:17:57.940230 4704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:17:57.944165 4704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:17:57.945641 4704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Loaded TSK: 0
I20260927 22:17:57.947329 4704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:17:57.950573 4694 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 22:17:58.305001 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:58.305531 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:58.305761 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:58.316601 4696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:58.316798 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:58.317028 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:17:58.332939 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=42447
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:58.334913 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:58.336738 4696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:58.363238 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:58.365021 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:58.380820 4696 server_base.cc:1034] running on GCE node
W20260927 22:17:58.381503 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:58.382431 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:58.385314 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:58.389382 4696 hybrid_clock.cc:648] HybridClock initialized: now 1790547478389314 us; error 55 us; skew 500 ppm
I20260927 22:17:58.389746 4696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:58 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:58.393473 4696 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:17:58.395627 4696 webserver.cc:460] Webserver started at http://127.23.173.65:42447/ using document root <none> and password file <none>
I20260927 22:17:58.396535 4696 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:58.396775 4696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:17:58 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:58.434870 4696 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.000s
I20260927 22:17:58.441368 4742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:58.443569 4696 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260927 22:17:58.443864 4696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "2393e79232c24b2fb0078dfef1468b09"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "3dd0a5b65778cf86531ef901f6203abb"
server_key_iv: "41c33dd908377647f71b6eeb007769f5"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:58.445410 4696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:58.510160 4696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:58.515554 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:58.515913 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:58.525269 4696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:58.533347 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:58.533519 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:17:58.533718 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:58.533874 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:58.601233 4696 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:42723
I20260927 22:17:58.601284 4855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:42723 every 8 connection(s)
I20260927 22:17:58.603309 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:17:58.610987 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4696
I20260927 22:17:58.611845 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:40235
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:35911
--webserver_port=38183
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:58 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:58.653126 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:58.617505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.65:43543)
0927 22:17:58.618373 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:58.618419 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:58.620106 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:58.625932 (+ 5826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:58.625945 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:58.626614 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:58.627447 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:58.627476 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:58.629918 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:58.629925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:58.632065 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:58.632081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:58.634064 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:58.634073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:58.634355 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:58.635168 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:58.635200 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:58.642968 (+ 7768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:58.647693 (+ 4725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:58.647702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:58.647718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:58.648358 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:58.648921 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:58.648927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:58.648931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:58.649214 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:58.650341 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:58.650350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:58.650841 (+ 491us) client_negotiation.cc:769] Sending connection context
0927 22:17:58.651144 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 22:17:58.651692 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":387,"threads_started":1}
I20260927 22:17:58.653277 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:58.617523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:43543 (local address 127.23.173.126:35911)
0927 22:17:58.624458 (+ 6935us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:58.624472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:58.624513 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:17:58.624825 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:58.624838 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:58.625170 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:58.625583 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:58.627687 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:58.629729 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:58.632396 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:58.633460 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:58.633646 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:58.643332 (+ 9686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:58.643344 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:58.643366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:58.643420 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:58.647502 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:58.648562 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:58.648572 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:58.648581 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:58.648789 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:58.649341 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:58.649346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:58.649350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:58.650048 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:58.650246 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:58.651434 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0927 22:17:58.651854 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6673,"thread_start_us":6440,"threads_started":1}
I20260927 22:17:58.656970 4856 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:58.657287 4856 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:58.658154 4856 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
W20260927 22:17:58.661396 4856 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:35911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:17:58.661615 4856 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:58.661888 4856 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:58.663081 4856 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:58.663329 4856 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
W20260927 22:17:58.666175 4856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:17:59.007133 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:59.007444 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:59.007618 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:59.016054 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:59.016213 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:59.016377 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:17:59.026528 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=38183
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:59.027860 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:59.029111 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:59.043766 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:59.043790 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:59.044404 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:59.046080 4860 server_base.cc:1034] running on GCE node
I20260927 22:17:59.046746 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:59.047888 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:59.049141 4860 hybrid_clock.cc:648] HybridClock initialized: now 1790547479049072 us; error 51 us; skew 500 ppm
I20260927 22:17:59.049497 4860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:59.053798 4860 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:17:59.056931 4860 webserver.cc:460] Webserver started at http://127.23.173.66:38183/ using document root <none> and password file <none>
I20260927 22:17:59.057785 4860 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:59.057941 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:59.090230 4860 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.000s
I20260927 22:17:59.101648 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:59.103615 4860 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.003s
I20260927 22:17:59.103853 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "ec32e167aea04eb0bb0a3201df86241a"
format_stamp: "Formatted at 2026-09-27 22:17:55 on dist-test-slave-3wc3"
server_key: "0782adde4e8d4f492b86aedb45acd17a"
server_key_iv: "adf475d0fc4158fe60da73a198ca45e0"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:59.105393 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:59.136169 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:59.141326 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:59.141611 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:59.142824 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:59.144682 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:59.144802 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:59.144913 4860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:59.144989 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:59.249797 4860 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:40235
I20260927 22:17:59.249833 4999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:40235 every 8 connection(s)
I20260927 22:17:59.252256 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:17:59.259788 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 4860
I20260927 22:17:59.260608 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.173.84:56263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:40559
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:35911
--webserver_port=38217
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:59.286290 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:59.256830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.66:42477)
0927 22:17:59.257689 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:59.257733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:59.260734 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:59.261644 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:59.261652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:59.262170 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:59.262766 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:59.262784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.263877 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.263882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:59.265817 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:59.265833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.267183 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.267191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:59.267495 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:59.268320 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:59.268349 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:59.271179 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:59.281265 (+ 10086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:59.281276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:59.281295 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:59.281950 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:59.282451 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:59.282456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:59.282460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:59.282754 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:59.283488 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:59.283497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:59.283948 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 22:17:59.284261 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 22:17:59.284856 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260927 22:17:59.287375 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:59.256208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:42477 (local address 127.23.173.126:35911)
0927 22:17:59.258707 (+ 2499us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:59.258713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:59.258739 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:17:59.261172 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:59.261176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:59.261252 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:59.261373 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:59.262944 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.263722 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.266046 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.266744 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.266871 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:59.271465 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:59.271472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:59.271477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:59.271518 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:59.281060 (+ 9542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.282144 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.282150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.282155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.282310 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.282892 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.282897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.282901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.283263 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:59.283399 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:59.286798 (+ 3399us) server_negotiation.cc:299] Negotiation successful
0927 22:17:59.287026 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":2266,"threads_started":1}
I20260927 22:17:59.289618 5000 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:59.290005 5000 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:59.290922 5000 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:59.293161 4659 ts_manager.cc:194] Registered new tserver with Master: ec32e167aea04eb0bb0a3201df86241a (127.23.173.66:40235)
I20260927 22:17:59.295441 4659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:42477
I20260927 22:17:59.666309 4856 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:59.666790 4856 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:59.667975 4659 ts_manager.cc:194] Registered new tserver with Master: 2393e79232c24b2fb0078dfef1468b09 (127.23.173.65:42723)
I20260927 22:17:59.669106 4659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:43543
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
W20260927 22:17:59.695842 5005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:17:59.696205 5005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:17:59.696439 5005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:17:59.707585 5005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:17:59.707799 5005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:17:59.708037 5005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:17:59.723217 5005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.173.84:56263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:40559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=38217
--tserver_master_addrs=127.23.173.126:35911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:17:59.725263 5005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:17:59.727108 5005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:17:59.772696 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:17:59.778705 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:59.794581 5005 server_base.cc:1034] running on GCE node
W20260927 22:17:59.795542 5013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:17:59.796442 5005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:17:59.805033 5005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:17:59.806277 5005 hybrid_clock.cc:648] HybridClock initialized: now 1790547479806220 us; error 61 us; skew 500 ppm
I20260927 22:17:59.806653 5005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:17:59.810259 5005 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:17:59.812076 5005 webserver.cc:460] Webserver started at http://127.23.173.67:38217/ using document root <none> and password file <none>
I20260927 22:17:59.812852 5005 fs_manager.cc:359] Metadata directory not provided
I20260927 22:17:59.813009 5005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:59.838248 5005 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:17:59.842136 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:59.843940 5005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:17:59.844108 5005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "52600e480ea64333b9eb18e647e3725e"
format_stamp: "Formatted at 2026-09-27 22:17:56 on dist-test-slave-3wc3"
server_key: "c4a522575abc2d0d12714869b6dff369"
server_key_iv: "169b4a3feb9f90a51b7758244d1c71e9"
server_key_version: "kuduclusterkey@0"
I20260927 22:17:59.845391 5005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:17:59.871474 5005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:17:59.876422 5005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:17:59.876744 5005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:17:59.877929 5005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:17:59.879631 5005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:17:59.879767 5005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:17:59.879899 5005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:17:59.879993 5005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:17:59.940431 5005 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:40559
I20260927 22:17:59.940513 5133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:40559 every 8 connection(s)
I20260927 22:17:59.942495 5005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:17:59.949275 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5005
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:17:59.981690 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:59.951383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.23.173.126:35911)
0927 22:17:59.958629 (+ 7246us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:59.958636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:59.958659 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:17:59.958716 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:59.958721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:59.958793 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:59.958928 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:59.960990 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.962109 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.964242 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.964922 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.965039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:59.969775 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:59.969781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:59.969785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:59.969823 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:59.977161 (+ 7338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.978017 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.978022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.978025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.978212 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.978631 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.978635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.978637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.979235 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:59.979490 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:59.981015 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0927 22:17:59.981285 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7105,"thread_start_us":6785,"threads_started":1}
Sep 27 22:17:59 dist-test-slave-3wc3 krb5kdc[3648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547479, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for HTTP/127.23.173.84@KRBTEST.COM
I20260927 22:17:59.986786 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:59.951143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:57237 (local address 127.23.173.126:35911)
0927 22:17:59.958379 (+ 7236us) server_negotiation.cc:206] Beginning negotiation
0927 22:17:59.958385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:17:59.958406 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:17:59.958469 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:17:59.958474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:17:59.958568 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:17:59.959088 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:17:59.965748 (+ 6660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.969618 (+ 3870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.972775 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.973411 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.973536 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:59.978038 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:17:59.978046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:17:59.978051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:17:59.978103 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:17:59.981444 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.983086 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.983091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.983094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.983450 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:17:59.984680 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:17:59.984685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:17:59.984689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:17:59.985030 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:17:59.985185 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:17:59.986158 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 22:17:59.986400 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7115,"thread_start_us":7173,"threads_started":1}
I20260927 22:17:59.988085 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:17:59.950330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:35911 (local address 127.23.173.67:57237)
0927 22:17:59.954457 (+ 4127us) negotiation.cc:107] Waiting for socket to connect
0927 22:17:59.954501 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:17:59.956866 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:17:59.962942 (+ 6076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:17:59.962956 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:17:59.963612 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:17:59.965488 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:59.965520 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.970412 (+ 4892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.970417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:59.972587 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:17:59.972604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:17:59.973912 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:17:59.973918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:17:59.974238 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:17:59.974957 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:17:59.974989 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:17:59.977726 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:17:59.981612 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:59.981622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:59.981641 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:59.982937 (+ 1296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:59.984109 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:17:59.984115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:17:59.984121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:17:59.984537 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:17:59.985211 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:17:59.985221 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:17:59.985670 (+ 449us) client_negotiation.cc:769] Sending connection context
0927 22:17:59.986041 (+ 371us) client_negotiation.cc:240] Negotiation successful
0927 22:17:59.986623 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3833,"thread_start_us":4124,"threads_started":1}
I20260927 22:17:59.989801 5134 heartbeater.cc:347] Connected to a master server at 127.23.173.126:35911
I20260927 22:17:59.990166 5134 heartbeater.cc:464] Registering TS with master...
I20260927 22:17:59.991067 5134 heartbeater.cc:511] Master 127.23.173.126:35911 requested a full tablet report, sending...
I20260927 22:17:59.992821 4659 ts_manager.cc:194] Registered new tserver with Master: 52600e480ea64333b9eb18e647e3725e (127.23.173.67:40559)
I20260927 22:17:59.993947 4659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:57237
I20260927 22:18:00.002044 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:18:00.093649 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.038630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.23.173.126:35911)
0927 22:18:00.038945 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.038952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.049177 (+ 10225us) server_negotiation.cc:407] Connection header received
0927 22:18:00.049667 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.049696 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.049798 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.049889 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:00.063495 (+ 13606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.064605 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.066700 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.067322 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.067418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.086848 (+ 19430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:00.086856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:00.086861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:00.086913 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:00.090473 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.091362 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.091368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.091371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.091511 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.091948 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.091953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.091956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.092308 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:00.092507 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.093087 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.093315 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260927 22:18:00.100792 4659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:18:00.112605 4659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:18:00.187950 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.154593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42723 (local address 127.0.0.1:33704)
0927 22:18:00.178292 (+ 23699us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:00.178312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:00.178496 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:00.179578 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:00.179583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:00.179605 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:00.180053 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.180064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.182016 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.182023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.184633 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.184648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.186841 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.186847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.187093 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.187160 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 22:18:00.187292 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:18:00.187500 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23546,"mutex_wait_us":34,"spinlock_wait_cycles":10880,"thread_start_us":21253,"threads_started":1}
I20260927 22:18:00.188884 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.155253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.23.173.65:42723)
0927 22:18:00.174262 (+ 19009us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.174275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.178750 (+ 4475us) server_negotiation.cc:407] Connection header received
0927 22:18:00.178876 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.178883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.179133 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.179330 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:00.180286 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.181833 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.184829 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.186310 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.186440 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.186674 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.188386 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.188557 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18792,"thread_start_us":18598,"threads_started":1}
I20260927 22:18:00.192478 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.154028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:40559 (local address 127.0.0.1:51988)
0927 22:18:00.176124 (+ 22096us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:00.176146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:00.176321 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:00.177891 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:00.177901 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:00.177935 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:00.178519 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.178538 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.182107 (+ 3569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.182114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.189350 (+ 7236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.189366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.191656 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.191662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.191843 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.191876 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:18:00.191956 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:18:00.192081 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21960,"mutex_wait_us":4545,"thread_start_us":23849,"threads_started":1}
I20260927 22:18:00.193342 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.154222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.23.173.67:40559)
0927 22:18:00.175642 (+ 21420us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.175653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.176677 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 22:18:00.176826 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.176838 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.177157 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.177423 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:00.179046 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.181938 (+ 2892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.189551 (+ 7613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.191182 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.191302 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.191520 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.192792 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.192986 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21208,"thread_start_us":22193,"threads_started":1}
I20260927 22:18:00.195182 4791 tablet_service.cc:1467] Processing CreateTablet for tablet 2eba2af007d143b4991355601c643757 (DEFAULT_TABLE table=test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:00.196830 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.151604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:40235 (local address 127.0.0.1:56884)
0927 22:18:00.158535 (+ 6931us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:00.158573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:00.158792 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:00.179634 (+ 20842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:00.179638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:00.179669 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:00.180154 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.180168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.188735 (+ 8567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.188741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.190813 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.190829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.196026 (+ 5197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.196032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.196235 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.196270 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:18:00.196367 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:18:00.196498 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6664,"thread_start_us":6596,"threads_started":1}
I20260927 22:18:00.196965 5069 tablet_service.cc:1467] Processing CreateTablet for tablet 2eba2af007d143b4991355601c643757 (DEFAULT_TABLE table=test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:00.198042 4791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eba2af007d143b4991355601c643757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:00.198628 5069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eba2af007d143b4991355601c643757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:00.203529 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.151812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.23.173.66:40235)
0927 22:18:00.179052 (+ 27240us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.179061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.179090 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:18:00.179195 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.179212 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.179437 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.179644 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:00.180365 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.188523 (+ 8158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.191054 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.194183 (+ 3129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.194335 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.194545 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.203021 (+ 8476us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.203215 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27074,"thread_start_us":26957,"threads_started":1}
I20260927 22:18:00.207087 4935 tablet_service.cc:1467] Processing CreateTablet for tablet 2eba2af007d143b4991355601c643757 (DEFAULT_TABLE table=test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:00.208425 4935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eba2af007d143b4991355601c643757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:00.233791 5156 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Bootstrap starting.
I20260927 22:18:00.235927 5157 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Bootstrap starting.
I20260927 22:18:00.237787 5156 tablet_bootstrap.cc:654] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:00.239511 5157 tablet_bootstrap.cc:654] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:00.240200 5158 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Bootstrap starting.
I20260927 22:18:00.243165 5157 log.cc:824] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:00.244886 5158 tablet_bootstrap.cc:654] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:00.246592 5158 log.cc:824] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:00.249470 5156 log.cc:824] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:00.255350 5158 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: No bootstrap required, opened a new log
I20260927 22:18:00.256578 5158 ts_tablet_manager.cc:1397] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260927 22:18:00.260020 5157 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: No bootstrap required, opened a new log
I20260927 22:18:00.260383 5157 ts_tablet_manager.cc:1397] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20260927 22:18:00.266089 5156 tablet_bootstrap.cc:492] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: No bootstrap required, opened a new log
I20260927 22:18:00.266412 5156 ts_tablet_manager.cc:1397] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.005s
I20260927 22:18:00.274487 5157 raft_consensus.cc:348] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.274984 5157 raft_consensus.cc:374] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:00.275100 5157 raft_consensus.cc:729] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2393e79232c24b2fb0078dfef1468b09, State: Initialized, Role: FOLLOWER
I20260927 22:18:00.275761 5158 raft_consensus.cc:348] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.275878 5157 consensus_queue.cc:260] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.276321 5158 raft_consensus.cc:374] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:00.276454 5158 raft_consensus.cc:729] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec32e167aea04eb0bb0a3201df86241a, State: Initialized, Role: FOLLOWER
I20260927 22:18:00.277210 5158 consensus_queue.cc:260] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.279528 5156 raft_consensus.cc:348] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.280059 5156 raft_consensus.cc:374] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:00.280247 5156 raft_consensus.cc:729] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52600e480ea64333b9eb18e647e3725e, State: Initialized, Role: FOLLOWER
I20260927 22:18:00.280856 4856 heartbeater.cc:503] Master 127.23.173.126:35911 was elected leader, sending a full tablet report...
I20260927 22:18:00.281354 5157 ts_tablet_manager.cc:1428] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Time spent starting tablet: real 0.021s user 0.011s sys 0.003s
I20260927 22:18:00.281073 5156 consensus_queue.cc:260] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.284057 5000 heartbeater.cc:503] Master 127.23.173.126:35911 was elected leader, sending a full tablet report...
I20260927 22:18:00.285058 5158 ts_tablet_manager.cc:1428] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Time spent starting tablet: real 0.028s user 0.015s sys 0.000s
I20260927 22:18:00.288993 5134 heartbeater.cc:503] Master 127.23.173.126:35911 was elected leader, sending a full tablet report...
I20260927 22:18:00.289565 5156 ts_tablet_manager.cc:1428] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Time spent starting tablet: real 0.023s user 0.014s sys 0.005s
W20260927 22:18:00.368058 4857 tablet.cc:2369] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:00.448604 5135 tablet.cc:2369] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:00.509725 5001 tablet.cc:2369] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:00.642266 5166 raft_consensus.cc:482] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:18:00.642536 5166 raft_consensus.cc:504] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.644268 5166 leader_election.cc:288] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52600e480ea64333b9eb18e647e3725e (127.23.173.67:40559), 2393e79232c24b2fb0078dfef1468b09 (127.23.173.65:42723)
I20260927 22:18:00.653611 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.644630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:50503 (local address 127.23.173.67:40559)
0927 22:18:00.644878 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.644888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.645684 (+ 796us) server_negotiation.cc:407] Connection header received
0927 22:18:00.645915 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.645920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.646037 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.646133 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:00.646815 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.647942 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.651133 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.652446 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.652612 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.652688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.653228 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.653366 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:18:00.653615 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.644794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:54487 (local address 127.23.173.65:42723)
0927 22:18:00.645075 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.645085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.645768 (+ 683us) server_negotiation.cc:407] Connection header received
0927 22:18:00.645992 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.645996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.646052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.646151 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:00.646947 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.648016 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.651139 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.652446 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.652618 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.652688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.653228 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.653362 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:18:00.654006 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.644772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:42723 (local address 127.23.173.66:54487)
0927 22:18:00.645662 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:00.645677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:00.645829 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:00.646167 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:00.646171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:00.646192 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:00.646675 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.646684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.648144 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.648149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.650975 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.650990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.652738 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.652743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.653013 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.653095 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 22:18:00.653186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:18:00.653349 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":314,"thread_start_us":549,"threads_started":1}
I20260927 22:18:00.654030 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.644532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:40559 (local address 127.23.173.66:50503)
0927 22:18:00.645564 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:00.645580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:00.645764 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:00.646147 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:00.646152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:00.646175 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:00.646649 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.646658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.648068 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.648095 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.650975 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:00.650990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.652720 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.652727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:00.653009 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.653087 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:18:00.653186 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:18:00.653349 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":364,"thread_start_us":663,"threads_started":1}
I20260927 22:18:00.655220 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eba2af007d143b4991355601c643757" candidate_uuid: "ec32e167aea04eb0bb0a3201df86241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2393e79232c24b2fb0078dfef1468b09" is_pre_election: true
I20260927 22:18:00.655321 5089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eba2af007d143b4991355601c643757" candidate_uuid: "ec32e167aea04eb0bb0a3201df86241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52600e480ea64333b9eb18e647e3725e" is_pre_election: true
I20260927 22:18:00.655746 5089 raft_consensus.cc:2454] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32e167aea04eb0bb0a3201df86241a in term 0.
I20260927 22:18:00.655746 4811 raft_consensus.cc:2454] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec32e167aea04eb0bb0a3201df86241a in term 0.
I20260927 22:18:00.656415 4888 leader_election.cc:302] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52600e480ea64333b9eb18e647e3725e, ec32e167aea04eb0bb0a3201df86241a; no voters:
I20260927 22:18:00.656843 5166 raft_consensus.cc:2781] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:18:00.656991 5166 raft_consensus.cc:482] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:18:00.657104 5166 raft_consensus.cc:3037] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:00.659354 5166 raft_consensus.cc:504] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.659965 5166 leader_election.cc:288] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [CANDIDATE]: Term 1 election: Requested vote from peers 52600e480ea64333b9eb18e647e3725e (127.23.173.67:40559), 2393e79232c24b2fb0078dfef1468b09 (127.23.173.65:42723)
I20260927 22:18:00.660338 5089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eba2af007d143b4991355601c643757" candidate_uuid: "ec32e167aea04eb0bb0a3201df86241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52600e480ea64333b9eb18e647e3725e"
I20260927 22:18:00.660452 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eba2af007d143b4991355601c643757" candidate_uuid: "ec32e167aea04eb0bb0a3201df86241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2393e79232c24b2fb0078dfef1468b09"
I20260927 22:18:00.660537 5089 raft_consensus.cc:3037] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:00.660682 4811 raft_consensus.cc:3037] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:00.662779 5089 raft_consensus.cc:2454] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32e167aea04eb0bb0a3201df86241a in term 1.
I20260927 22:18:00.662987 4811 raft_consensus.cc:2454] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec32e167aea04eb0bb0a3201df86241a in term 1.
I20260927 22:18:00.663168 4888 leader_election.cc:302] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52600e480ea64333b9eb18e647e3725e, ec32e167aea04eb0bb0a3201df86241a; no voters:
I20260927 22:18:00.663450 5166 raft_consensus.cc:2781] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:00.664006 5166 raft_consensus.cc:686] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 1 LEADER]: Becoming Leader. State: Replica: ec32e167aea04eb0bb0a3201df86241a, State: Running, Role: LEADER
I20260927 22:18:00.664489 5166 consensus_queue.cc:237] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } }
I20260927 22:18:00.668994 4658 catalog_manager.cc:5495] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec32e167aea04eb0bb0a3201df86241a (127.23.173.66). New cstate: current_term: 1 leader_uuid: "ec32e167aea04eb0bb0a3201df86241a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec32e167aea04eb0bb0a3201df86241a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 40235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20260927 22:18:00.749292 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.743440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.23.173.66:40235)
0927 22:18:00.744270 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.744274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.744288 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:18:00.744373 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.744376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.744454 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.744586 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:00.745020 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.745842 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.747193 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.747618 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.747709 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.748120 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:00.748628 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:18:00.748736 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.748926 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.749007 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":591,"threads_started":1}
I20260927 22:18:00.757321 4811 raft_consensus.cc:1261] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 1 FOLLOWER]: Refusing update from remote peer ec32e167aea04eb0bb0a3201df86241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:00.757345 5089 raft_consensus.cc:1261] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 1 FOLLOWER]: Refusing update from remote peer ec32e167aea04eb0bb0a3201df86241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:00.758332 5166 consensus_queue.cc:1035] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2393e79232c24b2fb0078dfef1468b09" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:00.758689 5174 consensus_queue.cc:1035] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52600e480ea64333b9eb18e647e3725e" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:00.769510 5182 mvcc.cc:205] Tried to move back new op lower bound from 7334082481172676608 to 7334082480805216256. Current Snapshot: MvccSnapshot[applied={T|T < 7334082481172676608}]
I20260927 22:18:00.769697 5183 mvcc.cc:205] Tried to move back new op lower bound from 7334082481172676608 to 7334082480805216256. Current Snapshot: MvccSnapshot[applied={T|T < 7334082481172676608}]
I20260927 22:18:00.784114 4659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:00.784343 4659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:00.787650 4659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5213b9ec87f8438b92c35d5adbdb814d: Sending DeleteTablet for 3 replicas of tablet 2eba2af007d143b4991355601c643757
I20260927 22:18:00.788753 4935 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eba2af007d143b4991355601c643757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:56884
I20260927 22:18:00.788942 5069 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eba2af007d143b4991355601c643757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:51988
I20260927 22:18:00.788929 4791 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eba2af007d143b4991355601c643757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33704
I20260927 22:18:00.792500 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4696
I20260927 22:18:00.801368 5194 tablet_replica.cc:330] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: stopping tablet replica
I20260927 22:18:00.802856 5192 tablet_replica.cc:330] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: stopping tablet replica
I20260927 22:18:00.803103 5194 raft_consensus.cc:2229] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:00.803431 5192 raft_consensus.cc:2229] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:18:00.804126 5194 raft_consensus.cc:2258] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:00.804126 5192 raft_consensus.cc:2258] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:00.804956 5193 tablet_replica.cc:330] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: stopping tablet replica
I20260927 22:18:00.805379 5193 raft_consensus.cc:2229] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:00.805852 5193 raft_consensus.cc:2258] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:00.806092 5192 ts_tablet_manager.cc:1905] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:00.807706 5194 ts_tablet_manager.cc:1905] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:00.807824 5193 ts_tablet_manager.cc:1905] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:00.813514 5192 ts_tablet_manager.cc:1918] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:00.813695 5192 log.cc:1196] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/2eba2af007d143b4991355601c643757
I20260927 22:18:00.814523 5192 ts_tablet_manager.cc:1939] T 2eba2af007d143b4991355601c643757 P ec32e167aea04eb0bb0a3201df86241a: Deleting consensus metadata
I20260927 22:18:00.815184 5193 ts_tablet_manager.cc:1918] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:00.815336 5193 log.cc:1196] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/2eba2af007d143b4991355601c643757
I20260927 22:18:00.815871 5193 ts_tablet_manager.cc:1939] T 2eba2af007d143b4991355601c643757 P 2393e79232c24b2fb0078dfef1468b09: Deleting consensus metadata
I20260927 22:18:00.816445 4644 catalog_manager.cc:4841] TS ec32e167aea04eb0bb0a3201df86241a (127.23.173.66:40235): tablet 2eba2af007d143b4991355601c643757 (table test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]) successfully deleted
I20260927 22:18:00.817623 4645 catalog_manager.cc:4841] TS 2393e79232c24b2fb0078dfef1468b09 (127.23.173.65:42723): tablet 2eba2af007d143b4991355601c643757 (table test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]) successfully deleted
I20260927 22:18:00.819906 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.793722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.23.173.65:42723)
0927 22:18:00.793930 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.793938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.802578 (+ 8640us) server_negotiation.cc:407] Connection header received
0927 22:18:00.802819 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.802825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.802907 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.803009 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:00.804860 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.806075 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.808195 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.808899 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.809049 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.812410 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:00.812429 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:00.812454 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:00.812503 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:00.816767 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.817627 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.817639 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.817642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.817796 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.818232 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.818238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.818242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.818626 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:00.818968 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.819293 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.819525 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:18:00.820554 4851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:00.821396 5194 ts_tablet_manager.cc:1918] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:00.821570 5194 log.cc:1196] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/2eba2af007d143b4991355601c643757
I20260927 22:18:00.822098 5194 ts_tablet_manager.cc:1939] T 2eba2af007d143b4991355601c643757 P 52600e480ea64333b9eb18e647e3725e: Deleting consensus metadata
I20260927 22:18:00.824055 4643 catalog_manager.cc:4841] TS 52600e480ea64333b9eb18e647e3725e (127.23.173.67:40559): tablet 2eba2af007d143b4991355601c643757 (table test-table [id=ef711ee6e7f04c6e8e80f7f8181df772]) successfully deleted
I20260927 22:18:00.928025 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4696
I20260927 22:18:00.948426 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4860
I20260927 22:18:00.966821 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:00.949266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.23.173.66:40235)
0927 22:18:00.949466 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:00.949472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:00.949597 (+ 125us) server_negotiation.cc:407] Connection header received
0927 22:18:00.949784 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:00.949789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:00.949870 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:00.950031 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:00.951778 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.953004 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.955019 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:00.955684 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:00.955802 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:00.959155 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:00.959171 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:00.959189 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:00.959229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:00.963614 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.964489 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.964503 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.964506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.964672 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:00.965099 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:00.965104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:00.965108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:00.965547 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:00.965859 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:00.966179 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 22:18:00.966436 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:18:00.967540 4995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:01.081718 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4860
I20260927 22:18:01.099737 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5005
I20260927 22:18:01.113029 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:01.100553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.23.173.67:40559)
0927 22:18:01.100749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:01.100753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:01.100773 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:18:01.100979 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:01.100983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:01.101046 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:01.101132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:01.102591 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:01.103331 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:01.104713 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:01.105114 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:01.105208 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:01.107774 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:01.107788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:01.107804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:01.107840 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:01.110703 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:01.111327 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:01.111334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:01.111336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:01.111454 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:01.111758 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:01.111761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:01.111763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:01.112098 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:01.112311 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:01.112512 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:18:01.112661 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:18:01.113746 5129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:01.205612 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5005
I20260927 22:18:01.222134 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 4626
I20260927 22:18:01.223379 4689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:01.334911 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 4626
I20260927 22:18:01.357107 24245 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:18:01.693020 24245 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:18:01.693218 24245 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:18:02.039093 24245 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:18:02.039 UTC [3678] LOG: received smart shutdown request
2026-09-27 22:18:02.042 UTC [3678] LOG: background worker "logical replication launcher" (PID 3687) exited with exit code 1
2026-09-27 22:18:02.042 UTC [3682] LOG: shutting down
2026-09-27 22:18:02.204 UTC [3678] LOG: database system is shut down
2026-09-27T22:18:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51586 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:18:02 dist-test-slave-3wc3 krb5kdc[5208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:18:02 dist-test-slave-3wc3 krb5kdc[5208](info): set up 2 sockets
Sep 27 22:18:02 dist-test-slave-3wc3 krb5kdc[5208](info): commencing operation
krb5kdc: starting...
W20260927 22:18:03.438493 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:18:03 dist-test-slave-3wc3 krb5kdc[5208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:18:03.661490 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:46847
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:03.986187 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:03.986472 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:03.986650 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:03.993798 5221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:03.993893 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:03.993937 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:03.994046 5221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:03.994151 5221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:04.004122 5221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:04.005434 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:04.006757 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:04.018373 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:04.018615 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:04.018677 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:04.019459 5221 server_base.cc:1034] running on GCE node
I20260927 22:18:04.020265 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:04.021709 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:04.022945 5221 hybrid_clock.cc:648] HybridClock initialized: now 1790547484022918 us; error 61 us; skew 500 ppm
I20260927 22:18:04.023304 5221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:04.026665 5221 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:04.028155 5221 webserver.cc:460] Webserver started at http://127.23.173.126:34459/ using document root <none> and password file <none>
I20260927 22:18:04.028788 5221 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:04.028903 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:04.029183 5221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:04.032135 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "b20260d4d7f14854811f4b7f09b50248"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "3c92d0c160da4ae02a4a9d6346b6d964"
server_key_iv: "de190321a29e6866785b99f2549a88fc"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.032927 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "b20260d4d7f14854811f4b7f09b50248"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "3c92d0c160da4ae02a4a9d6346b6d964"
server_key_iv: "de190321a29e6866785b99f2549a88fc"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.037932 5221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:18:04.041265 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:04.042634 5221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:04.042809 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "b20260d4d7f14854811f4b7f09b50248"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "3c92d0c160da4ae02a4a9d6346b6d964"
server_key_iv: "de190321a29e6866785b99f2549a88fc"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.043015 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:04.060328 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:04.064926 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:04.065227 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:04.085153 5221 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:46847
I20260927 22:18:04.085165 5287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:46847 every 8 connection(s)
I20260927 22:18:04.086824 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:04.090598 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:04.094007 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5221
I20260927 22:18:04.094245 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:18:04.095072 24245 external_mini_cluster.cc:1428] Setting key 16b8faeb4af060ca0060b7496c9cf34e
I20260927 22:18:04.099417 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.125@KRBTEST.COM" created.
I20260927 22:18:04.123624 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.103484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:46847 (local address 127.0.0.1:52310)
0927 22:18:04.104637 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.104676 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:04.106520 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:04.107333 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:04.107340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:04.108074 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:04.108404 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.108421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.109982 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.109990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.111051 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.111064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.112103 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.112111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.112282 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.112991 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:04.113012 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:04.116153 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:04.119612 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.119622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.119631 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.120113 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.120510 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.120514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.120516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.120662 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.121634 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:04.121640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:04.121829 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 22:18:04.122115 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:18:04.122508 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":721,"threads_started":1}
I20260927 22:18:04.123665 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.103484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.23.173.126:46847)
0927 22:18:04.104372 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:04.104391 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:04.104793 (+ 402us) server_negotiation.cc:407] Connection header received
0927 22:18:04.106880 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:04.106885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:04.107105 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:04.107314 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:04.108598 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.109768 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.111193 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.111838 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.112049 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.116387 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:04.116395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:04.116406 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:04.116435 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:04.119472 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.120239 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.120245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.120248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.120406 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.120803 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.120807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.120809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.121470 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:04.121675 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:04.122297 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 22:18:04.122711 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":629,"thread_start_us":390,"threads_started":1}
I20260927 22:18:04.128706 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:18:04.129927 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.129416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48062)
0927 22:18:04.129623 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.129760 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 22:18:04.130246 5237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
I20260927 22:18:04.131891 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.131434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48074)
0927 22:18:04.131604 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.131699 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:18:04.132099 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:35427: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
I20260927 22:18:04.149372 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:35427
--webserver_interface=127.23.173.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:04.181465 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } attempt: 1
I20260927 22:18:04.182739 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.182212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48078)
0927 22:18:04.182423 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.182557 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:18:04.183733 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.183359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48084)
0927 22:18:04.183501 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.183586 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:18:04.183936 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:35427: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
I20260927 22:18:04.252197 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } attempt: 2
I20260927 22:18:04.253470 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.252881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48090)
0927 22:18:04.253175 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.253291 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:18:04.254645 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.254142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48098)
0927 22:18:04.254328 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.254446 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:18:04.254874 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:35427: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
I20260927 22:18:04.346136 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } attempt: 3
I20260927 22:18:04.347400 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.346855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48100)
0927 22:18:04.347096 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.347234 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:18:04.348507 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.348080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48108)
0927 22:18:04.348234 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.348336 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:18:04.348697 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:35427: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
W20260927 22:18:04.476506 5295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:04.476790 5295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:04.476879 5295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:04.483991 5295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:04.484090 5295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:04.484135 5295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:04.484195 5295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:04.484238 5295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:04.491973 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } attempt: 4
I20260927 22:18:04.493538 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.492832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48116)
0927 22:18:04.493229 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.493387 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260927 22:18:04.494597 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.494249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48132)
0927 22:18:04.494394 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.494492 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:18:04.494805 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:35427: Network error: Client connection negotiation failed: client connection to 127.23.173.125:35427: connect: Connection refused (error 111)
I20260927 22:18:04.495606 5295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:04.496987 5295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:04.498340 5295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:04.510027 5301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:04.510078 5303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:04.510142 5300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:04.510839 5295 server_base.cc:1034] running on GCE node
I20260927 22:18:04.511510 5295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:04.512843 5295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:04.514065 5295 hybrid_clock.cc:648] HybridClock initialized: now 1790547484514016 us; error 41 us; skew 500 ppm
I20260927 22:18:04.514357 5295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:04.524333 5295 init.cc:377] Logged in from keytab as kudu/127.23.173.125@KRBTEST.COM (short username kudu)
I20260927 22:18:04.525784 5295 webserver.cc:460] Webserver started at http://127.23.173.125:34549/ using document root <none> and password file <none>
I20260927 22:18:04.526530 5295 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:04.526674 5295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:04.526984 5295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:04.529937 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data/instance:
uuid: "992ec9aff90b498e90753d3d6dc8adcd"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "88dcd079fb84fd61c7f94f8864ede476"
server_key_iv: "a2e9c6119d695180cea8d7014b448406"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.530920 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal/instance:
uuid: "992ec9aff90b498e90753d3d6dc8adcd"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "88dcd079fb84fd61c7f94f8864ede476"
server_key_iv: "a2e9c6119d695180cea8d7014b448406"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.535844 5295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:18:04.539218 5310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:04.540601 5295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:18:04.540769 5295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "992ec9aff90b498e90753d3d6dc8adcd"
format_stamp: "Formatted at 2026-09-27 22:18:04 on dist-test-slave-3wc3"
server_key: "88dcd079fb84fd61c7f94f8864ede476"
server_key_iv: "a2e9c6119d695180cea8d7014b448406"
server_key_version: "encryptionkey@0"
I20260927 22:18:04.540966 5295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:04.552772 5295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:04.556595 5295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:04.556881 5295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:04.575212 5295 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.125:35427
I20260927 22:18:04.575244 5361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.125:35427 every 8 connection(s)
I20260927 22:18:04.576581 5295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
I20260927 22:18:04.580240 5362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:04.580994 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5295
I20260927 22:18:04.581198 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-1/wal/instance
I20260927 22:18:04.581651 24245 external_mini_cluster.cc:1428] Setting key a2f6fa53d1aed74bedd365a24ec7ce5c
I20260927 22:18:04.586827 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.124@KRBTEST.COM" created.
I20260927 22:18:04.607209 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.589458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.23.173.126:46847)
0927 22:18:04.589714 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:04.589723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:04.590907 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 22:18:04.592773 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:04.592780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:04.592873 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:04.593005 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:04.594515 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.595644 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.597126 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.597536 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.597702 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.601351 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:04.601356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:04.601359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:04.601397 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:04.604185 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.604898 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.604905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.604913 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.605022 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.605469 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.605473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.605478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.605743 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:04.605892 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:04.606687 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 22:18:04.606901 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:18:04.608168 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.589902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:46847 (local address 127.0.0.1:52326)
0927 22:18:04.590735 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.590786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:04.592507 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:04.593181 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:04.593188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:04.593697 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:04.594318 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.594336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.595790 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.595794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.596973 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.596984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.597793 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.597800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.598059 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.598748 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:04.598769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:04.601093 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:04.604336 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.604344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.604360 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.604773 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.605134 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.605139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.605142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.605332 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.605879 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:04.605886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:04.606311 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 22:18:04.606579 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 22:18:04.607061 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":332,"threads_started":1}
I20260927 22:18:04.609946 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:18:04.626998 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.610672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48146)
0927 22:18:04.610835 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.610854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:04.610954 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:04.612225 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:04.612229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:04.612722 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:04.612856 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.612864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.614979 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.614982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.615806 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.615817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.617075 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.617079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.617272 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.617910 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:04.617927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:04.620173 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:04.624556 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.624560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.624562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.624918 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.625339 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.625342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.625345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.625495 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.626138 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:04.626144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:04.626265 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 22:18:04.626390 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:18:04.626702 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":2}
I20260927 22:18:04.627087 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.610974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.23.173.125:35427)
0927 22:18:04.611705 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:04.611715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:04.611748 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:18:04.611809 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:04.611820 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:04.612045 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:04.612220 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:04.613092 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.614842 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.615947 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.616920 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.617061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.620417 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:04.620435 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:04.620457 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:04.620503 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:04.624387 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.625057 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.625069 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.625075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.625221 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.625591 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.625595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.625597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.625944 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:04.626182 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:04.626534 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:18:04.626762 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":544,"thread_start_us":384,"threads_started":1}
I20260927 22:18:04.629842 5362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:18:04.631048 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.630654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52210)
0927 22:18:04.630801 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.630901 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 22:18:04.631400 5314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
I20260927 22:18:04.632894 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.632505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52216)
0927 22:18:04.632688 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.632761 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:18:04.633106 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.634014 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:34513
--webserver_interface=127.23.173.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:04.682417 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 1
I20260927 22:18:04.683557 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.683107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52222)
0927 22:18:04.683296 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.683411 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 22:18:04.684584 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.684248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52234)
0927 22:18:04.684423 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.684483 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:18:04.684836 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.753058 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 2
I20260927 22:18:04.754276 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.753789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52244)
0927 22:18:04.754030 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.754122 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:18:04.755271 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.754969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52250)
0927 22:18:04.755092 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.755155 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:18:04.755546 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.794056 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } attempt: 5
Sep 27 22:18:04 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:18:04.808825 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.795105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.23.173.125:35427)
0927 22:18:04.795310 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:04.795314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:04.795331 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:18:04.795393 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:04.795397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:04.795454 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:04.795538 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:04.796605 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.797270 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.799002 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.799458 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.799554 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.803339 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:04.803344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:04.803358 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:04.803389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:04.806006 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.806923 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.806927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.806929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.807049 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:04.807512 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:04.807514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:04.807516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:04.807778 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:04.807890 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:04.808394 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 22:18:04.808558 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:18:04.809003 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.794833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48160)
0927 22:18:04.795042 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.795064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:04.795209 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:04.795613 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:04.795618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:04.796192 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:04.796445 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.796458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.797409 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.797422 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.798847 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:04.798862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:04.799612 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:04.799618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:04.799806 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:04.800439 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:04.800461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:04.803145 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:04.806145 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.806153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.806159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.806803 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.807170 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:04.807179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:04.807184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:04.807380 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:04.807913 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:04.807920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:04.808129 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 22:18:04.808300 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 22:18:04.808627 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:18:04.810382 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:18:04.811501 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.811046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52266)
0927 22:18:04.811220 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.811308 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:18:04.812430 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.812060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52274)
0927 22:18:04.812201 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.812303 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:18:04.812629 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.846753 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 3
I20260927 22:18:04.847910 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.847415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52288)
0927 22:18:04.847615 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.847721 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:18:04.849013 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.848577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52302)
0927 22:18:04.848810 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.848897 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260927 22:18:04.849278 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.863886 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 1
I20260927 22:18:04.865146 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.864575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52318)
0927 22:18:04.864813 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.864932 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:18:04.866331 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.865866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52332)
0927 22:18:04.866082 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.866182 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:18:04.866549 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:04.934783 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 2
I20260927 22:18:04.936111 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.935471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52334)
0927 22:18:04.935740 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.935879 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:18:04.937206 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.936806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52350)
0927 22:18:04.936961 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.937050 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:18:04.937448 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
W20260927 22:18:04.968242 5369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:04.968530 5369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:04.968652 5369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:04.975826 5369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:04.975980 5369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:04.976073 5369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:04.976174 5369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:04.976255 5369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:04.986802 5369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:46847,127.23.173.125:35427,127.23.173.124:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:04.988221 5369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:04.989645 5369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:18:04.992522 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 4
I20260927 22:18:04.993656 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.993194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52352)
0927 22:18:04.993412 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.993499 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:18:04.994668 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:04.994340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52362)
0927 22:18:04.994490 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:04.994566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:18:04.994917 5362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
W20260927 22:18:05.002393 5374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:05.002429 5375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:05.002400 5377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:05.003198 5369 server_base.cc:1034] running on GCE node
I20260927 22:18:05.004025 5369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:05.005415 5369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:05.006608 5369 hybrid_clock.cc:648] HybridClock initialized: now 1790547485006523 us; error 86 us; skew 500 ppm
I20260927 22:18:05.006925 5369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:05.010152 5369 init.cc:377] Logged in from keytab as kudu/127.23.173.124@KRBTEST.COM (short username kudu)
I20260927 22:18:05.011660 5369 webserver.cc:460] Webserver started at http://127.23.173.124:40195/ using document root <none> and password file <none>
I20260927 22:18:05.012336 5369 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:05.012439 5369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:05.012696 5369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:05.015964 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data/instance:
uuid: "359a4ba71f2142128a9a32e05b69305e"
format_stamp: "Formatted at 2026-09-27 22:18:05 on dist-test-slave-3wc3"
server_key: "259605501334557f806c5379af637a92"
server_key_iv: "290b49f188ead917f37ef511d00068e5"
server_key_version: "encryptionkey@0"
I20260927 22:18:05.016741 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal/instance:
uuid: "359a4ba71f2142128a9a32e05b69305e"
format_stamp: "Formatted at 2026-09-27 22:18:05 on dist-test-slave-3wc3"
server_key: "259605501334557f806c5379af637a92"
server_key_iv: "290b49f188ead917f37ef511d00068e5"
server_key_version: "encryptionkey@0"
I20260927 22:18:05.022112 5369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:18:05.025692 5384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:05.027149 5369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:05.027346 5369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "359a4ba71f2142128a9a32e05b69305e"
format_stamp: "Formatted at 2026-09-27 22:18:05 on dist-test-slave-3wc3"
server_key: "259605501334557f806c5379af637a92"
server_key_iv: "290b49f188ead917f37ef511d00068e5"
server_key_version: "encryptionkey@0"
I20260927 22:18:05.027559 5369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:05.043756 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 3
I20260927 22:18:05.044970 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.044424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52368)
0927 22:18:05.044614 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.044745 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:18:05.046120 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.045666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52378)
0927 22:18:05.045915 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.046026 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260927 22:18:05.046326 5288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:34513: Network error: Client connection negotiation failed: client connection to 127.23.173.124:34513: connect: Connection refused (error 111)
I20260927 22:18:05.049124 5369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:05.053550 5369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:05.053838 5369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:05.072260 5369 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.124:34513
I20260927 22:18:05.072293 5435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.124:34513 every 8 connection(s)
I20260927 22:18:05.073583 5369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
I20260927 22:18:05.075609 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5369
I20260927 22:18:05.075810 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790547283249637-24245-0/minicluster-data/master-2/wal/instance
I20260927 22:18:05.076232 24245 external_mini_cluster.cc:1428] Setting key 0fbc2f7a391e7f55aa467953854950b8
I20260927 22:18:05.077935 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:05.087258 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:05.101030 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.078126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.23.173.124:34513)
0927 22:18:05.078891 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.078906 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.078967 (+ 61us) server_negotiation.cc:407] Connection header received
0927 22:18:05.081090 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.081118 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.081623 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.082097 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.083424 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.085463 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.087393 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.088457 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.088639 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.093851 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.093858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.093867 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.093900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.096816 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.097507 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.097513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.097519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.097688 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.098162 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.098166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.098168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.098642 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.098940 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.099334 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.099675 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260927 22:18:05.105705 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.090139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:46847 (local address 127.0.0.1:52336)
0927 22:18:05.091228 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.091269 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:05.091486 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:05.091889 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:05.091896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:05.092698 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:05.093057 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.093069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.094522 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.094527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.095701 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.095713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.096473 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.096478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.096669 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.097229 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:05.097246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:05.099478 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:05.102975 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.102985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.102990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.103551 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.103968 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.103975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.103980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.104146 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.104811 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:05.104817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:05.105041 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 22:18:05.105234 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:18:05.105447 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":606,"threads_started":1}
I20260927 22:18:05.106127 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.090403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.23.173.126:46847)
0927 22:18:05.090683 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.090690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.091358 (+ 668us) server_negotiation.cc:407] Connection header received
0927 22:18:05.091643 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.091648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.091709 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.091817 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.093222 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.094319 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.095863 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.096282 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.096427 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.099676 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.099682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.099686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.099723 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.102851 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.103674 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.103679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.103682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.103839 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.104252 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.104257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.104260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.104672 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.104892 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.105394 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.105691 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:18:05.107316 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:05.121160 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.108008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:35427 (local address 127.0.0.1:48172)
0927 22:18:05.108292 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.108305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:05.108424 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:05.108750 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:05.108755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:05.109110 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:05.109251 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.109263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.110390 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.110395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.111337 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.111349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.112253 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.112260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.112465 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.113049 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:05.113065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:05.115420 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:05.118819 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.118826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.118830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.119192 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.119573 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.119577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.119581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.119720 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.120222 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:05.120227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:05.120388 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 22:18:05.120528 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:18:05.120799 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 22:18:05.121186 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.108019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.23.173.125:35427)
0927 22:18:05.108199 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.108203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.108380 (+ 177us) server_negotiation.cc:407] Connection header received
0927 22:18:05.108555 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.108560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.108630 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.108742 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.109416 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.110253 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.111477 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.112110 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.112199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.115614 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.115622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.115629 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.115674 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.118687 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.119315 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.119321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.119325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.119461 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.119836 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.119848 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.119852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.120109 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.120273 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.120627 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.120827 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:18:05.122205 5436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:18:05.122953 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.104253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.23.173.126:46847)
0927 22:18:05.106445 (+ 2192us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.106457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.106479 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:18:05.106543 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.106551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.106639 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.106755 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.108910 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.110452 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.111976 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.112526 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.112664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.117502 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.117510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.117516 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.117557 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.120183 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.120859 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.120865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.120867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.120989 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.121446 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.121450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.121453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.121770 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.122082 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.122401 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.122610 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":5917,"threads_started":1}
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:18:05.135697 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.123060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52400)
0927 22:18:05.123281 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.123300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:05.123411 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:05.123740 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:05.123743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:05.124159 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:05.124330 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.124341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.125441 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.125449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.126358 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.126369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.127099 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.127104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.127279 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.127771 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:05.127793 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:05.130352 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:05.133259 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.133267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.133271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.133746 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.134221 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.134226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.134231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.134410 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.134900 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:05.134905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:05.135070 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 22:18:05.135194 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:18:05.135390 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:18:05.136054 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.123260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.23.173.124:34513)
0927 22:18:05.123410 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.123414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.123429 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:18:05.123538 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.123542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.123610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.123714 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.124502 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.125318 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.126531 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.126948 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.127085 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.130515 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.130519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.130522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.130549 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.133127 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.133883 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.133887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.133889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.134105 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.134530 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.134533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.134535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.134785 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.134900 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.135316 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.135568 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:18:05.140695 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.124784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.23.173.125:35427)
0927 22:18:05.124952 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.124959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.124994 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:18:05.125173 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.125178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.125247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.125346 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.126614 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.127301 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.129206 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.129859 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.130011 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.135067 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.135071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.135074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.135101 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.138134 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.138777 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.138782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.138786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.138931 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.139333 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.139339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.139342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.139600 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.139897 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.140158 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.140390 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:18:05.141053 5436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: Bootstrap starting.
I20260927 22:18:05.145110 5436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:05.145731 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:18:05.146452 5436 log.cc:824] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:05.149272 5436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: No bootstrap required, opened a new log
I20260927 22:18:05.162664 5436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.163208 5436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:05.163339 5436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359a4ba71f2142128a9a32e05b69305e, State: Initialized, Role: FOLLOWER
I20260927 22:18:05.164075 5436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.168483 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.169003 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.171931 5436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:05.178145 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.151458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.23.173.126:46847)
0927 22:18:05.151678 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.151688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.158345 (+ 6657us) server_negotiation.cc:407] Connection header received
0927 22:18:05.158607 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.158617 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.158728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.158869 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.161047 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.162590 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.165700 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.166687 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.166821 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.170786 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.170793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.170797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.170854 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.174406 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.175416 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.175425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.175431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.175641 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.176170 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.176177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.176182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.176578 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.176815 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.177447 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.177708 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:18:05.184878 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.151162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.23.173.124:34513)
0927 22:18:05.151596 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.151602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.158572 (+ 6970us) server_negotiation.cc:407] Connection header received
0927 22:18:05.163146 (+ 4574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.163155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.163265 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.163369 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.167177 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.168336 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.170577 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.171262 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.171385 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.176740 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.176750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.176757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.176809 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.181147 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.182070 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.182075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.182079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.182280 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.182747 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.182750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.182752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.183111 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.183379 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.184275 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.184510 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:18:05.185065 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.151007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.23.173.125:35427)
0927 22:18:05.151194 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.151201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.166984 (+ 15783us) server_negotiation.cc:407] Connection header received
0927 22:18:05.167153 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.167158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.167245 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.167347 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.169724 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.170884 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.173185 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.173863 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.174038 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.177421 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.177427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.177432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.177469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.181790 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.182698 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.182703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.182706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.182869 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.183321 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.183327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.183333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.183716 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.183945 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.184497 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.184720 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 22:18:05.193629 5461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:18:05.193758 5461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:18:05.189741 5369 master_runner.cc:186] Error getting master registration for 127.23.173.126:46847: OK, instance_id { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" instance_seqno: 1790547484078600 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:18:05.199350 5369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:18:05.223580 5288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 4
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
I20260927 22:18:05.238754 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.224160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52414)
0927 22:18:05.224363 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.224380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:05.224470 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:05.224779 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:05.224782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:05.225139 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:05.225296 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.225304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.226891 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.226895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.227928 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.227941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.229196 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.229202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.229405 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.230211 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:05.230229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:05.232608 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:05.236444 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.236448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.236451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.236795 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.237220 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.237223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.237225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.237358 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.238042 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:05.238045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:05.238169 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 22:18:05.238283 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:18:05.238485 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:18:05.239041 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.224249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.23.173.124:34513)
0927 22:18:05.224451 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.224462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.224482 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:18:05.224566 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.224573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.224651 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.224752 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.225440 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.226743 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.228077 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.228738 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.228895 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.232789 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.232796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.232801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.232838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.236316 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.236931 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.236937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.236940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.237099 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.237461 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.237466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.237470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.237840 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.238020 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.238421 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.238660 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:18:05.243902 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: Bootstrap starting.
I20260927 22:18:05.247126 5288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:05.248265 5288 log.cc:824] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:05.251103 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: No bootstrap required, opened a new log
I20260927 22:18:05.258921 5288 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.259245 5288 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:05.259377 5288 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20260d4d7f14854811f4b7f09b50248, State: Initialized, Role: FOLLOWER
I20260927 22:18:05.259919 5288 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.260809 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.261209 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.262051 5288 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:18:05.272696 5475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:18:05.272924 5475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:18:05.271498 5221 master_runner.cc:186] Error getting master registration for 127.23.173.125:35427: OK, instance_id { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" instance_seqno: 1790547484569801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:18:05.284884 5221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:18:05.294166 5362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } attempt: 5
Sep 27 22:18:05 dist-test-slave-3wc3 krb5kdc[5208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
W20260927 22:18:05.306092 5390 tablet.cc:2369] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:05.312793 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.294864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:34513 (local address 127.0.0.1:52418)
0927 22:18:05.295145 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:05.295164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:05.295291 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:05.295697 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:05.295702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:05.296250 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:05.296464 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.296477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.299164 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.299171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.300609 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:05.300626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.301662 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.301668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:05.301904 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.302641 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:05.302665 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:05.305552 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:05.309853 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.309859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.309864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.310429 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.310868 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:05.310874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:05.310878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:05.311080 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:05.311723 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:05.311729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:05.311903 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 22:18:05.312058 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:18:05.312385 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 22:18:05.313184 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:05.294969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.23.173.124:34513)
0927 22:18:05.295157 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:05.295164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:05.295222 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:18:05.295435 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:05.295441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:05.295525 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:05.295665 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:05.296644 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.297853 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.300781 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:05.301497 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:05.301640 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:05.305751 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:05.305758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:05.305763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:05.305802 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:05.309687 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.310567 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.310573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.310577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.310735 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:05.311212 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:05.311218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:05.311221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:05.311583 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:05.311756 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:05.312200 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 22:18:05.312471 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:18:05.318102 5242 tablet.cc:2369] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:05.320827 5362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: Bootstrap starting.
I20260927 22:18:05.325045 5362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:05.326431 5362 log.cc:824] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:05.329819 5362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: No bootstrap required, opened a new log
I20260927 22:18:05.342836 5362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.343317 5362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:05.343439 5362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992ec9aff90b498e90753d3d6dc8adcd, State: Initialized, Role: FOLLOWER
I20260927 22:18:05.344166 5362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.345296 5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.345836 5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.346674 5362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:05.371411 5295 master_runner.cc:263] No leader master found from master 992ec9aff90b498e90753d3d6dc8adcd
I20260927 22:18:05.371578 5295 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 22:18:05.377831 5489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:18:05.378113 5489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:18:05.509828 5464 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:18:05.510181 5464 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.511960 5464 leader_election.cc:288] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 992ec9aff90b498e90753d3d6dc8adcd (127.23.173.125:35427), 359a4ba71f2142128a9a32e05b69305e (127.23.173.124:34513)
I20260927 22:18:05.513252 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b20260d4d7f14854811f4b7f09b50248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a4ba71f2142128a9a32e05b69305e" is_pre_election: true
I20260927 22:18:05.513921 5411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20260d4d7f14854811f4b7f09b50248 in term 0.
I20260927 22:18:05.515008 5240 leader_election.cc:302] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359a4ba71f2142128a9a32e05b69305e, b20260d4d7f14854811f4b7f09b50248; no voters:
I20260927 22:18:05.516215 5337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b20260d4d7f14854811f4b7f09b50248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992ec9aff90b498e90753d3d6dc8adcd" is_pre_election: true
I20260927 22:18:05.516786 5337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20260d4d7f14854811f4b7f09b50248 in term 0.
I20260927 22:18:05.517679 5464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:18:05.518080 5464 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:18:05.518198 5464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:05.521617 5464 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
I20260927 22:18:05.523437 5337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b20260d4d7f14854811f4b7f09b50248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992ec9aff90b498e90753d3d6dc8adcd"
I20260927 22:18:05.523676 5337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:05.527356 5337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20260d4d7f14854811f4b7f09b50248 in term 1.
I20260927 22:18:05.531940 5237 leader_election.cc:302] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992ec9aff90b498e90753d3d6dc8adcd, b20260d4d7f14854811f4b7f09b50248; no voters:
I20260927 22:18:05.535384 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b20260d4d7f14854811f4b7f09b50248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359a4ba71f2142128a9a32e05b69305e"
I20260927 22:18:05.535627 5411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:05.539263 5411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20260d4d7f14854811f4b7f09b50248 in term 1.
I20260927 22:18:05.540078 5464 leader_election.cc:288] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [CANDIDATE]: Term 1 election: Requested vote from peers 992ec9aff90b498e90753d3d6dc8adcd (127.23.173.125:35427), 359a4ba71f2142128a9a32e05b69305e (127.23.173.124:34513)
I20260927 22:18:05.540490 5464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:05.540918 5464 raft_consensus.cc:686] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [term 1 LEADER]: Becoming Leader. State: Replica: b20260d4d7f14854811f4b7f09b50248, State: Running, Role: LEADER
I20260927 22:18:05.541752 5464 consensus_queue.cc:237] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } }
W20260927 22:18:05.559919 5316 tablet.cc:2369] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:05.563841 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20260d4d7f14854811f4b7f09b50248. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.564409 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:05.569906 5493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:05.572546 5493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:05.612481 5411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [term 1 FOLLOWER]: Refusing update from remote peer b20260d4d7f14854811f4b7f09b50248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:05.613724 5491 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:05.621799 5337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [term 1 FOLLOWER]: Refusing update from remote peer b20260d4d7f14854811f4b7f09b50248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:05.622875 5491 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:05.646644 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20260d4d7f14854811f4b7f09b50248. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.647001 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.662463 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.662910 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:05.663674 5494 mvcc.cc:205] Tried to move back new op lower bound from 7334082501055639552 to 7334082500792197120. Current Snapshot: MvccSnapshot[applied={T|T < 7334082501055639552}]
I20260927 22:18:05.665007 5493 catalog_manager.cc:1347] Generated new cluster ID: dc6261bbaf074fe9b863a3f3447ffc14
I20260927 22:18:05.665108 5493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:05.666059 5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20260d4d7f14854811f4b7f09b50248. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.666435 5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.673134 5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.673388 5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.672485 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.677114 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:05.699973 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b20260d4d7f14854811f4b7f09b50248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20260d4d7f14854811f4b7f09b50248" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46847 } } peers { permanent_uuid: "992ec9aff90b498e90753d3d6dc8adcd" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 35427 } } peers { permanent_uuid: "359a4ba71f2142128a9a32e05b69305e" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 34513 } } }
I20260927 22:18:05.700261 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:18:05.701826 5493 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:18:05.703727 5493 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:05.709838 5496 mvcc.cc:205] Tried to move back new op lower bound from 7334082501055639552 to 7334082500792197120. Current Snapshot: MvccSnapshot[applied={T|T < 7334082501055639552 or (T in {7334082501055639552})}]
I20260927 22:18:05.727741 5493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b20260d4d7f14854811f4b7f09b50248: Generated new TSK 0
I20260927 22:18:05.728971 5493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:18:06.195801 5461 catalog_manager.cc:1259] Loaded cluster ID: dc6261bbaf074fe9b863a3f3447ffc14
I20260927 22:18:06.195950 5461 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: loading cluster ID for follower catalog manager: success
I20260927 22:18:06.198520 5461 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: acquiring CA information for follower catalog manager: success
I20260927 22:18:06.200279 5461 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 359a4ba71f2142128a9a32e05b69305e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:18:06.380118 5489 catalog_manager.cc:1259] Loaded cluster ID: dc6261bbaf074fe9b863a3f3447ffc14
I20260927 22:18:06.380231 5489 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: loading cluster ID for follower catalog manager: success
I20260927 22:18:06.382719 5489 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: acquiring CA information for follower catalog manager: success
I20260927 22:18:06.384492 5489 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 992ec9aff90b498e90753d3d6dc8adcd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:18:06.836359 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5221
I20260927 22:18:06.837643 5283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:06.948342 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5221
I20260927 22:18:06.969419 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5295
I20260927 22:18:06.970670 5357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:07.078639 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5295
I20260927 22:18:07.099041 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5369
I20260927 22:18:07.100152 5431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:07.207249 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5369
2026-09-27T22:18:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4873 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:18:07 dist-test-slave-3wc3 krb5kdc[5512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:18:07 dist-test-slave-3wc3 krb5kdc[5512](info): set up 2 sockets
Sep 27 22:18:07 dist-test-slave-3wc3 krb5kdc[5512](info): commencing operation
krb5kdc: starting...
W20260927 22:18:08.310456 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:18:08 dist-test-slave-3wc3 krb5kdc[5512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:18:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:18:08.545816 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39721
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:39721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:08.872890 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:08.873186 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:08.873353 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:08.880517 5525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:08.880625 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:08.880745 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:08.880851 5525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:08.880939 5525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:08.891091 5525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:39721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:08.892454 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:08.893787 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:08.905726 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:08.905787 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:08.905707 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:08.907281 5525 server_base.cc:1034] running on GCE node
I20260927 22:18:08.908036 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:08.909387 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:08.910624 5525 hybrid_clock.cc:648] HybridClock initialized: now 1790547488910574 us; error 55 us; skew 500 ppm
I20260927 22:18:08.910966 5525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:08 dist-test-slave-3wc3 krb5kdc[5512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547488, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:08.914003 5525 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:08.915414 5525 webserver.cc:460] Webserver started at http://127.23.173.126:44105/ using document root <none> and password file <none>
I20260927 22:18:08.916110 5525 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:08.916258 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:08.916579 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:08.919600 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "e2d03c56e12e4c23886980ef2550916f"
format_stamp: "Formatted at 2026-09-27 22:18:08 on dist-test-slave-3wc3"
server_key: "265c3a01d621ce6609ace634b45152e3"
server_key_iv: "ea80de2289ca8c020b2cad463eea3ac4"
server_key_version: "encryptionkey@0"
I20260927 22:18:08.920341 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "e2d03c56e12e4c23886980ef2550916f"
format_stamp: "Formatted at 2026-09-27 22:18:08 on dist-test-slave-3wc3"
server_key: "265c3a01d621ce6609ace634b45152e3"
server_key_iv: "ea80de2289ca8c020b2cad463eea3ac4"
server_key_version: "encryptionkey@0"
I20260927 22:18:08.925544 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:18:08.928951 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:08.930298 5525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:08.930487 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "e2d03c56e12e4c23886980ef2550916f"
format_stamp: "Formatted at 2026-09-27 22:18:08 on dist-test-slave-3wc3"
server_key: "265c3a01d621ce6609ace634b45152e3"
server_key_iv: "ea80de2289ca8c020b2cad463eea3ac4"
server_key_version: "encryptionkey@0"
I20260927 22:18:08.930687 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:08.954510 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:08.958508 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:08.958801 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:08.977547 5525 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:39721
I20260927 22:18:08.977567 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:39721 every 8 connection(s)
I20260927 22:18:08.979007 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:08.982637 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:08.988684 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5525
I20260927 22:18:08.988919 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:18:08.989614 24245 external_mini_cluster.cc:1428] Setting key 0c76102bfc0be44c2386cc1e9e7b78c9
I20260927 22:18:08.993256 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f: Bootstrap starting.
I20260927 22:18:08.996887 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:08.998028 5592 log.cc:824] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:09.001019 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f: No bootstrap required, opened a new log
Sep 27 22:18:08 dist-test-slave-3wc3 krb5kdc[5512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:09.010159 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } }
I20260927 22:18:09.010548 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:09.010656 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d03c56e12e4c23886980ef2550916f, State: Initialized, Role: FOLLOWER
I20260927 22:18:09.011348 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } }
I20260927 22:18:09.011626 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:09.011801 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:09.012017 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:09.013669 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:08.991600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.23.173.126:39721)
0927 22:18:08.992431 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:08.992445 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:08.992486 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:18:08.994310 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:08.994333 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:08.994730 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:08.995064 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:08.996899 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:08.998376 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:09.000158 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:09.000786 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:09.000911 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:09.005837 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:09.005845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:09.005861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:09.005898 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:09.009205 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:09.010031 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:09.010045 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:09.010052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:09.010229 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:09.010782 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:09.010790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:09.010793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:09.011338 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:09.011674 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:09.012122 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 22:18:09.012577 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":390,"threads_started":1}
I20260927 22:18:09.014621 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } }
I20260927 22:18:09.015118 5592 leader_election.cc:302] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d03c56e12e4c23886980ef2550916f; no voters:
I20260927 22:18:09.015848 5592 leader_election.cc:288] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:18:09.015997 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:09.017344 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [term 1 LEADER]: Becoming Leader. State: Replica: e2d03c56e12e4c23886980ef2550916f, State: Running, Role: LEADER
I20260927 22:18:09.017643 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:18:09.017992 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } }
I20260927 22:18:09.018394 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:09.021090 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d03c56e12e4c23886980ef2550916f. Latest consensus state: current_term: 1 leader_uuid: "e2d03c56e12e4c23886980ef2550916f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } } }
I20260927 22:18:09.021183 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d03c56e12e4c23886980ef2550916f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d03c56e12e4c23886980ef2550916f" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39721 } } }
I20260927 22:18:09.021505 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:09.021577 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:09.022518 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:09.029302 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:09.038695 5602 catalog_manager.cc:1347] Generated new cluster ID: 669264ecfd47472c82c7ac412298d37d
I20260927 22:18:09.038806 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:09.066157 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:18:09.068048 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:09.074479 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d03c56e12e4c23886980ef2550916f: Generated new TSK 0
I20260927 22:18:09.075258 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:18:09.129855 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5525
I20260927 22:18:09.131057 5587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:09.228442 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5525
2026-09-27T22:18:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2026 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:18:09 dist-test-slave-3wc3 krb5kdc[5620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:18:09 dist-test-slave-3wc3 krb5kdc[5620](info): set up 2 sockets
Sep 27 22:18:09 dist-test-slave-3wc3 krb5kdc[5620](info): commencing operation
krb5kdc: starting...
W20260927 22:18:10.334324 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:18:10 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:18:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:18:10.574411 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:10.897517 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:10.897856 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:10.898049 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:10.905295 5633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:10.905397 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:10.905516 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:10.905630 5633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:10.905714 5633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:10.915642 5633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:10.917022 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:10.918366 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:10.929889 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:10.929934 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:10.929984 5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:10.930836 5633 server_base.cc:1034] running on GCE node
I20260927 22:18:10.931748 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:10.933105 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:10.934338 5633 hybrid_clock.cc:648] HybridClock initialized: now 1790547490934307 us; error 60 us; skew 500 ppm
I20260927 22:18:10.934674 5633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:10 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:10.943948 5633 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:10.945528 5633 webserver.cc:460] Webserver started at http://127.23.173.126:38615/ using document root <none> and password file <none>
I20260927 22:18:10.946264 5633 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:10.946414 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:10.946717 5633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:10.949625 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "294b88e0d6f947e19948909a0e07b11a"
format_stamp: "Formatted at 2026-09-27 22:18:10 on dist-test-slave-3wc3"
server_key: "8c2affedd2a8f9d7959f62f0b451289d"
server_key_iv: "9e81d888405c36b4f5f9a5cb4cb0d1f7"
server_key_version: "encryptionkey@0"
I20260927 22:18:10.950404 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "294b88e0d6f947e19948909a0e07b11a"
format_stamp: "Formatted at 2026-09-27 22:18:10 on dist-test-slave-3wc3"
server_key: "8c2affedd2a8f9d7959f62f0b451289d"
server_key_iv: "9e81d888405c36b4f5f9a5cb4cb0d1f7"
server_key_version: "encryptionkey@0"
I20260927 22:18:10.955399 5633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:18:10.958732 5648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:10.960006 5633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:18:10.960187 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "294b88e0d6f947e19948909a0e07b11a"
format_stamp: "Formatted at 2026-09-27 22:18:10 on dist-test-slave-3wc3"
server_key: "8c2affedd2a8f9d7959f62f0b451289d"
server_key_iv: "9e81d888405c36b4f5f9a5cb4cb0d1f7"
server_key_version: "encryptionkey@0"
I20260927 22:18:10.960381 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:10.980861 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:10.984803 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:10.985087 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:11.003398 5633 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37429
I20260927 22:18:11.003433 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37429 every 8 connection(s)
I20260927 22:18:11.004737 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:11.006901 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5633
I20260927 22:18:11.007124 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:18:11.007931 24245 external_mini_cluster.cc:1428] Setting key a600d5c7f882d3fdbfb548da9e7b02b7
I20260927 22:18:11.009123 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:11.021257 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap starting.
Sep 27 22:18:11 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:11.024777 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:11.025833 5700 log.cc:824] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:11.028755 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: No bootstrap required, opened a new log
I20260927 22:18:11.029873 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:11.010128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.23.173.126:37429)
0927 22:18:11.010898 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:11.010911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:11.010949 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:18:11.012471 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:11.012491 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:11.012872 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:11.013283 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:11.014726 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.016034 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.017355 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.018088 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.018323 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:11.022577 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:11.022585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:11.022601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:11.022640 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:11.025686 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:11.026501 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:11.026525 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:11.026531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:11.026699 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:11.027108 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:11.027117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:11.027119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:11.027608 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:11.027952 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:11.028346 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:18:11.028777 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
I20260927 22:18:11.037014 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:11.037423 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:11.037540 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Initialized, Role: FOLLOWER
I20260927 22:18:11.038178 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:11.038481 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:11.038653 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:11.038847 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:11.040982 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:11.041510 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 294b88e0d6f947e19948909a0e07b11a; no voters:
I20260927 22:18:11.042161 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:18:11.042304 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:11.043027 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 LEADER]: Becoming Leader. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Running, Role: LEADER
I20260927 22:18:11.043545 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:11.044309 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:11.046790 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:11.047389 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:11.046895 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 294b88e0d6f947e19948909a0e07b11a. Latest consensus state: current_term: 1 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:11.047780 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:11.055174 5713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:11.060343 5713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:11.069514 5713 catalog_manager.cc:1347] Generated new cluster ID: 28ea3f8fbd6e4f9a8ef400f24872ec1f
I20260927 22:18:11.069643 5713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:11.094959 5713 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:18:11.096293 5713 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:11.107015 5713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Generated new TSK 0
I20260927 22:18:11.107831 5713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:18:11.175688 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:11.515648 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:11.515933 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:11.516077 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:11.525435 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:11.525548 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:11.525687 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:18:11.536689 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:11.538282 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:11.539605 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:11.553036 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:11.553009 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:11.553233 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:11.553874 5726 server_base.cc:1034] running on GCE node
I20260927 22:18:11.554646 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:11.555819 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:11.557044 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790547491556991 us; error 45 us; skew 500 ppm
I20260927 22:18:11.557464 5726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:11 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:11.561486 5726 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:18:11.563042 5726 webserver.cc:460] Webserver started at http://127.23.173.65:36659/ using document root <none> and password file <none>
I20260927 22:18:11.563841 5726 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:11.564005 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:11.564344 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:11.567687 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "767caf9a07da46e09bd21cfddd451a30"
format_stamp: "Formatted at 2026-09-27 22:18:11 on dist-test-slave-3wc3"
server_key: "866339fa80e7c3dc510414c10e3bff29"
server_key_iv: "7fe6e102a71737a030fad6bbcf597128"
server_key_version: "encryptionkey@0"
I20260927 22:18:11.568555 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "767caf9a07da46e09bd21cfddd451a30"
format_stamp: "Formatted at 2026-09-27 22:18:11 on dist-test-slave-3wc3"
server_key: "866339fa80e7c3dc510414c10e3bff29"
server_key_iv: "7fe6e102a71737a030fad6bbcf597128"
server_key_version: "encryptionkey@0"
I20260927 22:18:11.574198 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:18:11.577697 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:11.579182 5726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:11.579358 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "767caf9a07da46e09bd21cfddd451a30"
format_stamp: "Formatted at 2026-09-27 22:18:11 on dist-test-slave-3wc3"
server_key: "866339fa80e7c3dc510414c10e3bff29"
server_key_iv: "7fe6e102a71737a030fad6bbcf597128"
server_key_version: "encryptionkey@0"
I20260927 22:18:11.579555 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:11.592790 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:11.596985 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:11.597265 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:11.598209 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:11.599458 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:11.599578 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:11.599723 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:11.599797 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:11.632423 5726 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40439
I20260927 22:18:11.632465 5853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40439 every 8 connection(s)
I20260927 22:18:11.633713 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:18:11.642472 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5726
I20260927 22:18:11.642691 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:18:11.643519 24245 external_mini_cluster.cc:1428] Setting key ac4913d0aacde9f67b2e3eeb2411d503
Sep 27 22:18:11 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547491, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:11.655117 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:11.636301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:47857 (local address 127.23.173.126:37429)
0927 22:18:11.636739 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:11.636743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:11.637698 (+ 955us) server_negotiation.cc:407] Connection header received
0927 22:18:11.639502 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:11.639507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:11.639572 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:11.639670 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:11.641051 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.641763 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.644209 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.644724 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.644888 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:11.649242 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:11.649248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:11.649251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:11.649283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:11.651838 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:11.652760 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:11.652764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:11.652766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:11.652876 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:11.653342 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:11.653345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:11.653347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:11.653601 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:11.653787 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:11.654512 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 22:18:11.654748 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260927 22:18:11.656070 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:11.636854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:47857)
0927 22:18:11.637534 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:11.637570 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:11.639251 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:11.639865 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:11.639872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:11.640351 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:11.640903 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:11.640920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.641897 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.641901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:11.644035 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:11.644050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:11.644981 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:11.644989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:11.645266 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:11.646062 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:11.646090 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:11.649016 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:11.651980 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:11.651992 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:11.652010 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:11.652643 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:11.652981 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:11.652989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:11.652995 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:11.653233 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:11.653734 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:11.653744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:11.654137 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 22:18:11.654389 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:18:11.654895 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":255,"threads_started":1}
I20260927 22:18:11.657655 5854 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:11.658020 5854 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:11.658919 5854 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:11.661087 5665 ts_manager.cc:194] Registered new tserver with Master: 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439)
I20260927 22:18:11.662580 5665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:47857
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:18:11.697728 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:12.025738 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:12.026078 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:12.026239 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:12.033727 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:12.033896 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:12.034092 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:18:12.043923 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:12.045341 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:12.046789 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:12.062325 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:12.062479 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:12.062816 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:12.063449 5860 server_base.cc:1034] running on GCE node
I20260927 22:18:12.064107 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:12.065152 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:12.066401 5860 hybrid_clock.cc:648] HybridClock initialized: now 1790547492066328 us; error 49 us; skew 500 ppm
I20260927 22:18:12.066731 5860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:12 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547492, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:12.069855 5860 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:18:12.071214 5860 webserver.cc:460] Webserver started at http://127.23.173.66:41873/ using document root <none> and password file <none>
I20260927 22:18:12.071885 5860 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:12.072029 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:12.072332 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:12.075300 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "f04703c4d11d4055aedadc66736bfc88"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "88b5a2e2e1c502ed83710dcbbbeead6c"
server_key_iv: "a0a1b3bfa5f8cbe30c58a94ccd021db8"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.076073 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "f04703c4d11d4055aedadc66736bfc88"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "88b5a2e2e1c502ed83710dcbbbeead6c"
server_key_iv: "a0a1b3bfa5f8cbe30c58a94ccd021db8"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.081234 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:18:12.084936 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.086371 5860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:12.086549 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "f04703c4d11d4055aedadc66736bfc88"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "88b5a2e2e1c502ed83710dcbbbeead6c"
server_key_iv: "a0a1b3bfa5f8cbe30c58a94ccd021db8"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.086730 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:12.099356 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:12.103754 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:12.104053 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:12.105031 5860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:12.106418 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:12.106494 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.106618 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:12.106669 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.139240 5860 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34331
I20260927 22:18:12.139281 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34331 every 8 connection(s)
I20260927 22:18:12.140646 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:18:12.150485 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5860
I20260927 22:18:12.150696 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:18:12.151293 24245 external_mini_cluster.cc:1428] Setting key a29f88c8cbef28c7a95b27e191c48746
Sep 27 22:18:12 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547492, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:12.163278 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.143207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36071 (local address 127.23.173.126:37429)
0927 22:18:12.143388 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.143394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.144731 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 22:18:12.146739 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.146745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.146849 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.146970 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:12.148443 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.149165 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.150853 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.151546 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.151694 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.156102 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:12.156108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:12.156112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:12.156142 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:12.159878 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.160658 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.160666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.160672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.160839 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.161265 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.161272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.161279 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.161661 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:12.161815 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.162580 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.162863 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:18:12.163889 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.143759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:36071)
0927 22:18:12.144572 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.144610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.146477 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.147196 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.147204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.147710 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:12.148267 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.148294 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.149327 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.149337 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.150683 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.150694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.151801 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.151816 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.152018 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.152797 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:12.152819 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:12.155855 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:12.160032 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:12.160039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:12.160052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:12.160513 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:12.160944 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:12.160947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:12.160949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:12.161149 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:12.161799 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:12.161805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:12.162170 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.162403 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.162800 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":343,"threads_started":1}
I20260927 22:18:12.165326 5988 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:12.165644 5988 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:12.166464 5988 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:12.167927 5665 ts_manager.cc:194] Registered new tserver with Master: f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331)
I20260927 22:18:12.168766 5665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:36071
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:18:12.206229 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:12.533495 5993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:12.533784 5993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:12.534024 5993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:12.541332 5993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:12.541467 5993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:12.541651 5993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:18:12.551671 5993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:12.552970 5993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:12.554368 5993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:12.567735 6001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:12.567757 5999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:12.567821 5998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:12.568888 5993 server_base.cc:1034] running on GCE node
I20260927 22:18:12.569507 5993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:12.570500 5993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:12.571727 5993 hybrid_clock.cc:648] HybridClock initialized: now 1790547492571691 us; error 47 us; skew 500 ppm
I20260927 22:18:12.572027 5993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:12 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547492, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:12.575201 5993 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:18:12.576823 5993 webserver.cc:460] Webserver started at http://127.23.173.67:38069/ using document root <none> and password file <none>
I20260927 22:18:12.577592 5993 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:12.577754 5993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:12.578167 5993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:12.581286 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "d242fff3b64c42178e19230ae5e379f4"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "e8a95114ed2846a9a4e2d1ab1247c679"
server_key_iv: "4d150ab826c7e6022a5eb92634611dda"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.582273 5993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "d242fff3b64c42178e19230ae5e379f4"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "e8a95114ed2846a9a4e2d1ab1247c679"
server_key_iv: "4d150ab826c7e6022a5eb92634611dda"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.587953 5993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:18:12.591804 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.593263 5993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:18:12.593403 5993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "d242fff3b64c42178e19230ae5e379f4"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "e8a95114ed2846a9a4e2d1ab1247c679"
server_key_iv: "4d150ab826c7e6022a5eb92634611dda"
server_key_version: "encryptionkey@0"
I20260927 22:18:12.593582 5993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:12.619396 5993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:12.624166 5993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:12.624432 5993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:12.625379 5993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:12.626762 5993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:12.626840 5993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.626946 5993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:12.626994 5993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:12.663640 5993 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43393
I20260927 22:18:12.663647 6120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43393 every 8 connection(s)
I20260927 22:18:12.665221 5993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:18:12.666378 5854 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:12.668598 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 5993
I20260927 22:18:12.668807 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:18:12.669322 24245 external_mini_cluster.cc:1428] Setting key c2837b3ec7026c838ec8fb81386dec53
Sep 27 22:18:12 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547492, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:12.687034 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.667928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:44855 (local address 127.23.173.126:37429)
0927 22:18:12.668383 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.668390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.669581 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 22:18:12.671775 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.671783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.671902 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.672016 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:12.673541 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.674471 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.676226 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.676686 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.676855 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.680625 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:12.680631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:12.680635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:12.680670 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:12.684112 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.684873 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.684878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.684882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.685057 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.685505 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.685509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.685512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.685784 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:12.685902 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.686611 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.686775 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260927 22:18:12.688035 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.668555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:44855)
0927 22:18:12.669417 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.669463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.671456 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.672281 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.672295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.672806 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:12.673367 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.673384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.674663 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.674669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.676062 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.676076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.676955 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.676963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.677243 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.678040 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:12.678077 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:12.680397 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:12.684280 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:12.684288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:12.684304 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:12.684758 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:12.685176 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:12.685180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:12.685182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:12.685357 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:12.685911 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:12.685917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:12.686280 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.686512 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.686898 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":362,"threads_started":1}
I20260927 22:18:12.689296 6121 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:12.689606 6121 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:12.690436 6121 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:12.691816 5665 ts_manager.cc:194] Registered new tserver with Master: d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393)
I20260927 22:18:12.692632 5665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:44855
I20260927 22:18:12.697260 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:18:12.714661 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.701772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.23.173.126:37429)
0927 22:18:12.701944 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.701951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.702396 (+ 445us) server_negotiation.cc:407] Connection header received
0927 22:18:12.702604 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.702609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.702678 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.702799 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:12.704155 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.704909 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.706359 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.706770 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.706909 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.709374 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:12.709379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:12.709382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:12.709415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:12.711961 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.712619 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.712626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.712635 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.712750 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:12.713147 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:12.713151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:12.713155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:12.713485 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:12.713743 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.714093 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.714338 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:18:12.718612 5665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:18:12.725457 5665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:18:12.761041 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.738011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40439 (local address 127.0.0.1:59248)
0927 22:18:12.749476 (+ 11465us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.749538 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.749790 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.751807 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.751815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.751856 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:12.753438 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.753452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.755400 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.755406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.757704 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.757719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.759959 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.759964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.760179 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.760238 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.760387 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.760583 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11137,"mutex_wait_us":556,"thread_start_us":12615,"threads_started":1}
I20260927 22:18:12.761353 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.751086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34331 (local address 127.0.0.1:33322)
0927 22:18:12.752514 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.752538 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.752660 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.753729 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.753734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.753759 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:12.754312 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.754324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.756051 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.756058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.758272 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.758288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.760272 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.760277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.760561 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.760726 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.760810 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.760956 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":135,"thread_start_us":1131,"threads_started":1}
I20260927 22:18:12.761543 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.738554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.23.173.67:43393)
0927 22:18:12.750466 (+ 11912us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.750475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.751171 (+ 696us) server_negotiation.cc:407] Connection header received
0927 22:18:12.751274 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.751280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.751524 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.751695 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:12.753169 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.755113 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.758219 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.759692 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.759827 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.760071 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.760933 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.761126 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11712,"thread_start_us":11546,"threads_started":1}
I20260927 22:18:12.761740 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.738295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.23.173.65:40439)
0927 22:18:12.750835 (+ 12540us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.750844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.750877 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:18:12.751002 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.751010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.751301 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.751501 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:12.753615 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.755256 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.757925 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.759389 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.759513 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.759778 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.760428 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.760609 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12355,"thread_start_us":12199,"threads_started":1}
I20260927 22:18:12.762148 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.751357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.23.173.66:34331)
0927 22:18:12.753146 (+ 1789us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.753155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.753181 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:18:12.753270 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.753276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.753523 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.753732 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:12.754477 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.755872 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.758453 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.759823 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.759939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.760097 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.761591 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.761779 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":1452,"threads_started":1}
I20260927 22:18:12.762540 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.738383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:43393 (local address 127.0.0.1:55932)
0927 22:18:12.750894 (+ 12511us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.750919 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.751047 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.751995 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.752001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.752030 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:12.752951 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.752963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.755257 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.755262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.758032 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.758046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.760260 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.760267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.760561 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.760723 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.760810 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.762129 (+ 1319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12375,"mutex_wait_us":1376,"thread_start_us":10605,"threads_started":1}
I20260927 22:18:12.765899 6056 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc (DEFAULT_TABLE table=test-table [id=ebf504ef4df4482ebf1beef760e1dba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:12.765945 5923 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc (DEFAULT_TABLE table=test-table [id=ebf504ef4df4482ebf1beef760e1dba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:12.765939 5789 tablet_service.cc:1467] Processing CreateTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc (DEFAULT_TABLE table=test-table [id=ebf504ef4df4482ebf1beef760e1dba6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:12.767203 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa8a77d59e4eefb92426fb3099bcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:12.767309 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa8a77d59e4eefb92426fb3099bcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:12.767310 5923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42fa8a77d59e4eefb92426fb3099bcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:12.778864 6140 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Bootstrap starting.
I20260927 22:18:12.782357 6140 tablet_bootstrap.cc:654] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:12.783708 6140 log.cc:824] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:12.785097 6142 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Bootstrap starting.
I20260927 22:18:12.789732 6142 tablet_bootstrap.cc:654] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:12.791364 6142 log.cc:824] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:12.793437 6141 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Bootstrap starting.
I20260927 22:18:12.798069 6141 tablet_bootstrap.cc:654] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:12.799521 6141 log.cc:824] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:12.800947 6140 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: No bootstrap required, opened a new log
I20260927 22:18:12.800997 6142 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: No bootstrap required, opened a new log
I20260927 22:18:12.801277 6140 ts_tablet_manager.cc:1397] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20260927 22:18:12.801326 6142 ts_tablet_manager.cc:1397] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260927 22:18:12.802977 6141 tablet_bootstrap.cc:492] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: No bootstrap required, opened a new log
I20260927 22:18:12.803220 6141 ts_tablet_manager.cc:1397] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 22:18:12.808534 6142 raft_consensus.cc:348] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.808894 6142 raft_consensus.cc:374] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:12.808996 6142 raft_consensus.cc:729] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04703c4d11d4055aedadc66736bfc88, State: Initialized, Role: FOLLOWER
I20260927 22:18:12.809595 6142 consensus_queue.cc:260] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.810702 5988 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:12.811231 6142 ts_tablet_manager.cc:1428] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 22:18:12.812032 6140 raft_consensus.cc:348] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.812434 6140 raft_consensus.cc:374] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:12.812525 6140 raft_consensus.cc:729] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d242fff3b64c42178e19230ae5e379f4, State: Initialized, Role: FOLLOWER
I20260927 22:18:12.813097 6140 consensus_queue.cc:260] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.814821 6121 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:12.814817 6140 ts_tablet_manager.cc:1428] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260927 22:18:12.815035 6141 raft_consensus.cc:348] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.815444 6141 raft_consensus.cc:374] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:12.815541 6141 raft_consensus.cc:729] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767caf9a07da46e09bd21cfddd451a30, State: Initialized, Role: FOLLOWER
I20260927 22:18:12.816056 6141 consensus_queue.cc:260] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.817617 6141 ts_tablet_manager.cc:1428] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260927 22:18:12.886068 5855 tablet.cc:2369] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:12.892608 5989 tablet.cc:2369] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:12.917095 6122 tablet.cc:2369] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:12.957317 6147 raft_consensus.cc:482] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:18:12.957648 6147 raft_consensus.cc:504] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.959300 6147 leader_election.cc:288] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439), f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331)
I20260927 22:18:12.965229 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.959767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:37881 (local address 127.23.173.65:40439)
0927 22:18:12.960013 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.960019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.960042 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:18:12.960129 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.960138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.960233 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.960352 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:12.960901 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.961621 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.963441 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.964339 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.964525 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.964596 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.964852 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.964979 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:18:12.965344 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.959601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40439 (local address 127.23.173.67:37881)
0927 22:18:12.959820 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.959842 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.960012 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.960558 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.960561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.960595 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:12.960745 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.960753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.961762 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.961767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.963256 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.963267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.964499 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.964523 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.964697 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.964720 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.964827 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.964984 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:18:12.965894 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.960331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:43513 (local address 127.23.173.66:34331)
0927 22:18:12.960573 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.960579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.960775 (+ 196us) server_negotiation.cc:407] Connection header received
0927 22:18:12.960976 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.960982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.961050 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.961147 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:12.961638 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.962393 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.964129 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.965017 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.965187 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.965255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:12.965569 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:18:12.965714 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:18:12.966435 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.960009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34331 (local address 127.23.173.67:43513)
0927 22:18:12.960679 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:12.960692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:12.960832 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:12.961165 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:12.961169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:12.961192 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:12.961495 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.961503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.962542 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.962547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.963980 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:12.963992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.965208 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.965217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:12.965391 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.965420 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:18:12.966098 (+ 678us) client_negotiation.cc:240] Negotiation successful
0927 22:18:12.966211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":330,"threads_started":1}
I20260927 22:18:12.966439 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa8a77d59e4eefb92426fb3099bcdc" candidate_uuid: "d242fff3b64c42178e19230ae5e379f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767caf9a07da46e09bd21cfddd451a30" is_pre_election: true
I20260927 22:18:12.967032 5809 raft_consensus.cc:2454] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d242fff3b64c42178e19230ae5e379f4 in term 0.
I20260927 22:18:12.967460 5943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa8a77d59e4eefb92426fb3099bcdc" candidate_uuid: "d242fff3b64c42178e19230ae5e379f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04703c4d11d4055aedadc66736bfc88" is_pre_election: true
I20260927 22:18:12.967765 6010 leader_election.cc:302] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767caf9a07da46e09bd21cfddd451a30, d242fff3b64c42178e19230ae5e379f4; no voters:
I20260927 22:18:12.968016 5943 raft_consensus.cc:2454] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d242fff3b64c42178e19230ae5e379f4 in term 0.
I20260927 22:18:12.968183 6147 raft_consensus.cc:2781] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:18:12.968350 6147 raft_consensus.cc:482] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:18:12.968463 6147 raft_consensus.cc:3037] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:12.970817 6147 raft_consensus.cc:504] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.971400 6147 leader_election.cc:288] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [CANDIDATE]: Term 1 election: Requested vote from peers 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439), f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331)
I20260927 22:18:12.971796 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa8a77d59e4eefb92426fb3099bcdc" candidate_uuid: "d242fff3b64c42178e19230ae5e379f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767caf9a07da46e09bd21cfddd451a30"
I20260927 22:18:12.971881 5943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42fa8a77d59e4eefb92426fb3099bcdc" candidate_uuid: "d242fff3b64c42178e19230ae5e379f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04703c4d11d4055aedadc66736bfc88"
I20260927 22:18:12.971997 5809 raft_consensus.cc:3037] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:12.972110 5943 raft_consensus.cc:3037] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:12.974366 5809 raft_consensus.cc:2454] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d242fff3b64c42178e19230ae5e379f4 in term 1.
I20260927 22:18:12.974531 5943 raft_consensus.cc:2454] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d242fff3b64c42178e19230ae5e379f4 in term 1.
I20260927 22:18:12.974785 6010 leader_election.cc:302] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767caf9a07da46e09bd21cfddd451a30, d242fff3b64c42178e19230ae5e379f4; no voters:
I20260927 22:18:12.975054 6147 raft_consensus.cc:2781] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:12.975483 6147 raft_consensus.cc:686] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 1 LEADER]: Becoming Leader. State: Replica: d242fff3b64c42178e19230ae5e379f4, State: Running, Role: LEADER
I20260927 22:18:12.975934 6147 consensus_queue.cc:237] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } }
I20260927 22:18:12.980479 5663 catalog_manager.cc:5495] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d242fff3b64c42178e19230ae5e379f4 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "d242fff3b64c42178e19230ae5e379f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } health_report { overall_health: UNKNOWN } } }
I20260927 22:18:13.000723 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:12.995372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.23.173.67:43393)
0927 22:18:12.995595 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:12.995601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:12.995626 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:18:12.995813 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:12.995817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:12.995900 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:12.996018 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:12.996337 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.997047 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.998657 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:12.999095 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:12.999190 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:12.999617 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:13.000180 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:18:13.000289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:13.000364 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:18:13.000435 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:18:13.009310 5943 raft_consensus.cc:1261] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Refusing update from remote peer d242fff3b64c42178e19230ae5e379f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:13.009310 5809 raft_consensus.cc:1261] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Refusing update from remote peer d242fff3b64c42178e19230ae5e379f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:13.010375 6150 consensus_queue.cc:1035] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:13.010802 6147 consensus_queue.cc:1035] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:13.027899 6154 mvcc.cc:205] Tried to move back new op lower bound from 7334082531355115520 to 7334082531233427456. Current Snapshot: MvccSnapshot[applied={T|T < 7334082531355115520}]
I20260927 22:18:13.044587 5664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:13.044909 5664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:13.048712 5664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Sending DeleteTablet for 3 replicas of tablet 42fa8a77d59e4eefb92426fb3099bcdc
I20260927 22:18:13.049844 5789 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:13 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:59248
I20260927 22:18:13.049851 6056 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:13 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:55932
I20260927 22:18:13.050057 5923 tablet_service.cc:1514] Processing DeleteTablet for tablet 42fa8a77d59e4eefb92426fb3099bcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:13 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33322
I20260927 22:18:13.052392 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5726
I20260927 22:18:13.067678 6165 tablet_replica.cc:330] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: stopping tablet replica
I20260927 22:18:13.068336 6165 raft_consensus.cc:2229] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:13.069018 6165 raft_consensus.cc:2258] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:13.070356 6164 tablet_replica.cc:330] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: stopping tablet replica
I20260927 22:18:13.071445 6165 ts_tablet_manager.cc:1905] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:13.071419 6164 raft_consensus.cc:2229] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:13.071959 6164 raft_consensus.cc:2258] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:13.073655 6163 tablet_replica.cc:330] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: stopping tablet replica
I20260927 22:18:13.074126 6163 raft_consensus.cc:2229] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:18:13.074869 6163 raft_consensus.cc:2258] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:13.075569 6164 ts_tablet_manager.cc:1905] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:18:13 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:18:13.079203 6165 ts_tablet_manager.cc:1918] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:13.079458 6165 log.cc:1196] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/42fa8a77d59e4eefb92426fb3099bcdc
I20260927 22:18:13.080062 6165 ts_tablet_manager.cc:1939] T 42fa8a77d59e4eefb92426fb3099bcdc P f04703c4d11d4055aedadc66736bfc88: Deleting consensus metadata
I20260927 22:18:13.080345 6163 ts_tablet_manager.cc:1905] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:13.082036 5650 catalog_manager.cc:4841] TS f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331): tablet 42fa8a77d59e4eefb92426fb3099bcdc (table test-table [id=ebf504ef4df4482ebf1beef760e1dba6]) successfully deleted
I20260927 22:18:13.082798 6164 ts_tablet_manager.cc:1918] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:13.082988 6164 log.cc:1196] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/42fa8a77d59e4eefb92426fb3099bcdc
I20260927 22:18:13.083572 6164 ts_tablet_manager.cc:1939] T 42fa8a77d59e4eefb92426fb3099bcdc P 767caf9a07da46e09bd21cfddd451a30: Deleting consensus metadata
I20260927 22:18:13.086714 5650 catalog_manager.cc:4841] TS 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439): tablet 42fa8a77d59e4eefb92426fb3099bcdc (table test-table [id=ebf504ef4df4482ebf1beef760e1dba6]) successfully deleted
I20260927 22:18:13.090082 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:13.053526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.23.173.65:40439)
0927 22:18:13.053722 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:13.053728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:13.059491 (+ 5763us) server_negotiation.cc:407] Connection header received
0927 22:18:13.060763 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:13.060773 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:13.060896 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:13.061002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:13.066213 (+ 5211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.067859 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.071193 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.071892 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.072036 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:13.082427 (+ 10391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:13.082445 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:13.082476 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:13.082530 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:13.086527 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.087476 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.087486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.087490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.087701 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.088175 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.088181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.088185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.088663 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:13.088970 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:13.089346 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 22:18:13.089594 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:18:13.090878 5849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:13.091995 6163 ts_tablet_manager.cc:1918] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:13.092206 6163 log.cc:1196] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/42fa8a77d59e4eefb92426fb3099bcdc
I20260927 22:18:13.092881 6163 ts_tablet_manager.cc:1939] T 42fa8a77d59e4eefb92426fb3099bcdc P d242fff3b64c42178e19230ae5e379f4: Deleting consensus metadata
I20260927 22:18:13.095114 5651 catalog_manager.cc:4841] TS d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393): tablet 42fa8a77d59e4eefb92426fb3099bcdc (table test-table [id=ebf504ef4df4482ebf1beef760e1dba6]) successfully deleted
I20260927 22:18:13.201238 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5726
I20260927 22:18:13.225605 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5860
Sep 27 22:18:13 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:18:13.242874 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:13.226454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.23.173.66:34331)
0927 22:18:13.226641 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:13.226645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:13.226746 (+ 101us) server_negotiation.cc:407] Connection header received
0927 22:18:13.226921 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:13.226925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:13.226996 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:13.227079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:13.228795 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.229526 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.231593 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.231968 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.232060 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:13.237110 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:13.237124 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:13.237139 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:13.237180 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:13.240096 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.240793 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.240801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.240803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.240987 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.241418 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.241422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.241424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.241826 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:13.242100 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:13.242411 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:18:13.242583 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:18:13.243556 5983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:13.386184 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5860
I20260927 22:18:13.403855 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5993
Sep 27 22:18:13 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:18:13.422621 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:13.404706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.23.173.67:43393)
0927 22:18:13.404900 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:13.404905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:13.404997 (+ 92us) server_negotiation.cc:407] Connection header received
0927 22:18:13.405181 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:13.405185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:13.405272 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:13.405353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:13.407248 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.408187 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.410122 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:13.410567 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:13.410669 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:13.416344 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:13.416356 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:13.416372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:13.416410 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:13.419652 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.420485 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.420492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.420494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.420633 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:13.421042 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:13.421045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:13.421047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:13.421408 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:13.421625 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:13.422066 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 22:18:13.422237 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:18:13.423481 6116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:13.550097 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5993
I20260927 22:18:13.570709 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 5633
I20260927 22:18:13.571792 5695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:13.682498 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 5633
I20260927 22:18:13.704164 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--webserver_interface=127.23.173.126
--webserver_port=38615
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:14.029675 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:14.030042 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:14.030225 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:14.037581 6171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:14.037688 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:14.037796 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:14.037904 6171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:14.037983 6171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:14.047737 6171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=38615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:14.049050 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:14.050386 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:14.080127 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:14.080080 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:14.080149 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:14.083302 6171 server_base.cc:1034] running on GCE node
I20260927 22:18:14.084288 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:14.085940 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:14.087234 6171 hybrid_clock.cc:648] HybridClock initialized: now 1790547494087181 us; error 59 us; skew 500 ppm
I20260927 22:18:14.087636 6171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:14 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547494, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:14.091223 6171 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:14.092947 6171 webserver.cc:460] Webserver started at http://127.23.173.126:38615/ using document root <none> and password file <none>
I20260927 22:18:14.093715 6171 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:14.093878 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:14.100073 6171 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:18:14.102960 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:14.104382 6171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:18:14.104568 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "294b88e0d6f947e19948909a0e07b11a"
format_stamp: "Formatted at 2026-09-27 22:18:10 on dist-test-slave-3wc3"
server_key: "8c2affedd2a8f9d7959f62f0b451289d"
server_key_iv: "9e81d888405c36b4f5f9a5cb4cb0d1f7"
server_key_version: "encryptionkey@0"
I20260927 22:18:14.105607 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:14.126428 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:14.131052 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:14.131348 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:14.150691 6171 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37429
I20260927 22:18:14.150728 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37429 every 8 connection(s)
I20260927 22:18:14.152266 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:14.157099 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6171
I20260927 22:18:14.157924 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:40439
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=36659
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:14.159391 6239 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:18:14.162710 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap starting.
I20260927 22:18:14.182551 6239 log.cc:824] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:14.194922 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:18:14.195467 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap complete.
I20260927 22:18:14.205030 6239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:14.205874 6239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Initialized, Role: FOLLOWER
I20260927 22:18:14.206547 6239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:14.206801 6239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:14.206926 6239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:14.207116 6239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:18:14.211282 6239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:14.211803 6239 leader_election.cc:302] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 294b88e0d6f947e19948909a0e07b11a; no voters:
I20260927 22:18:14.212481 6239 leader_election.cc:288] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:18:14.212612 6243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:18:14.213976 6243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 LEADER]: Becoming Leader. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Running, Role: LEADER
I20260927 22:18:14.214576 6243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:14.214788 6239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:14.217276 6245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 294b88e0d6f947e19948909a0e07b11a. Latest consensus state: current_term: 2 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:14.217649 6245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:14.217371 6244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:14.217784 6244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:14.228132 6249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:14.238152 6249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:14.240012 6249 catalog_manager.cc:1259] Loaded cluster ID: 28ea3f8fbd6e4f9a8ef400f24872ec1f
I20260927 22:18:14.240195 6249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:14.244299 6249 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:18:14.269477 6249 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:18:14.280995 6249 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:14.283365 6249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Loaded TSK: 0
I20260927 22:18:14.285310 6249 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:18:14.525918 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:14.526244 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:14.526392 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:14.535020 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:14.535125 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:14.535311 6241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:18:14.545424 6241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=36659
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:14.546780 6241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:14.548117 6241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:14.563285 6266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:14.563558 6267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:14.564100 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:14.564438 6241 server_base.cc:1034] running on GCE node
I20260927 22:18:14.565161 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:14.566344 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:14.567606 6241 hybrid_clock.cc:648] HybridClock initialized: now 1790547494567537 us; error 62 us; skew 500 ppm
I20260927 22:18:14.567997 6241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:14 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547494, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:14.571918 6241 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:18:14.573498 6241 webserver.cc:460] Webserver started at http://127.23.173.65:36659/ using document root <none> and password file <none>
I20260927 22:18:14.574259 6241 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:14.574402 6241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:14.580111 6241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:18:14.583046 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:14.584506 6241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:14.584707 6241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "767caf9a07da46e09bd21cfddd451a30"
format_stamp: "Formatted at 2026-09-27 22:18:11 on dist-test-slave-3wc3"
server_key: "866339fa80e7c3dc510414c10e3bff29"
server_key_iv: "7fe6e102a71737a030fad6bbcf597128"
server_key_version: "encryptionkey@0"
I20260927 22:18:14.585703 6241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:14.607573 6241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:14.612334 6241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:14.612632 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:14.613656 6241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:14.615253 6241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:14.615374 6241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:14.615523 6241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:14.615653 6241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:18:14.650305 6241 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40439
I20260927 22:18:14.650365 6388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40439 every 8 connection(s)
I20260927 22:18:14.651867 6241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:18:14.661679 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6241
I20260927 22:18:14.662474 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:34331
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=41873
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:18:14 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547494, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:14.676441 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:14.654822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:34105 (local address 127.23.173.126:37429)
0927 22:18:14.655461 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:14.655473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:14.656191 (+ 718us) server_negotiation.cc:407] Connection header received
0927 22:18:14.658305 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:14.658318 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:14.658628 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:14.658938 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:14.660289 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:14.661506 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:14.663980 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:14.664570 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:14.664699 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:14.668905 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:14.668913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:14.668926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:14.668970 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:14.671865 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:14.672825 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:14.672831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:14.672837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:14.673018 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:14.673505 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:14.673508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:14.673510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:14.674063 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:14.674223 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:14.675259 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 22:18:14.675573 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
I20260927 22:18:14.676782 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:14.655127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:34105)
0927 22:18:14.656027 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:14.656073 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:14.657796 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:14.659068 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:14.659076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:14.659562 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:14.660092 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:14.660108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:14.661688 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:14.661694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:14.663832 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:14.663852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:14.664817 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:14.664826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:14.665144 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:14.665880 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:14.665918 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:14.668595 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:14.672028 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:14.672040 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:14.672058 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:14.672672 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:14.673136 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:14.673144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:14.673150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:14.673383 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:14.674194 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:14.674205 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:14.674643 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 22:18:14.674933 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 22:18:14.675498 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":375,"threads_started":1}
I20260927 22:18:14.679687 6389 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:14.680050 6389 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:14.680900 6389 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:14.683151 6204 ts_manager.cc:194] Registered new tserver with Master: 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439)
I20260927 22:18:14.684650 6204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:34105
W20260927 22:18:14.988657 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:14.988978 6393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:14.989166 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:14.996325 6393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:14.996474 6393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:14.996663 6393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:18:15.006472 6393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:34331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=41873
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:15.007786 6393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:15.009114 6393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:15.022465 6401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:15.022467 6399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:15.022706 6398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:15.023219 6393 server_base.cc:1034] running on GCE node
I20260927 22:18:15.023836 6393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:15.024724 6393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:15.026012 6393 hybrid_clock.cc:648] HybridClock initialized: now 1790547495025916 us; error 56 us; skew 500 ppm
I20260927 22:18:15.026340 6393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:15 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:15.029404 6393 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:18:15.030795 6393 webserver.cc:460] Webserver started at http://127.23.173.66:41873/ using document root <none> and password file <none>
I20260927 22:18:15.031443 6393 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:15.031548 6393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:15.036607 6393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:18:15.039331 6408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.040616 6393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:18:15.040745 6393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "f04703c4d11d4055aedadc66736bfc88"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "88b5a2e2e1c502ed83710dcbbbeead6c"
server_key_iv: "a0a1b3bfa5f8cbe30c58a94ccd021db8"
server_key_version: "encryptionkey@0"
I20260927 22:18:15.041653 6393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:15.065543 6393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:15.069490 6393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:15.069697 6393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:15.070585 6393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:15.071780 6393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:15.071869 6393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.071946 6393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:15.072002 6393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.104563 6393 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34331
I20260927 22:18:15.104655 6520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34331 every 8 connection(s)
I20260927 22:18:15.106048 6393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:18:15.109460 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6393
I20260927 22:18:15.110224 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:43393
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=38069
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:18:15 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:15.126153 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.108869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34595 (local address 127.23.173.126:37429)
0927 22:18:15.109063 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.109069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.110342 (+ 1273us) server_negotiation.cc:407] Connection header received
0927 22:18:15.112234 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.112242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.112349 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.112461 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:15.113912 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.114752 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.116389 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.116807 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.116938 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.120589 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:15.120593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:15.120597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:15.120630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:15.123246 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.123932 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.123939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.123943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.124069 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.124475 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.124481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.124485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.124755 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:15.124910 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.125567 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.125791 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:18:15.127148 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.109330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:34595)
0927 22:18:15.110184 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.110228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.111957 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.112702 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.112710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.113187 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:15.113733 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.113749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.114904 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.114910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.116234 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.116245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.117035 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.117042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.117265 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.117927 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:15.117949 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:15.120363 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:15.123388 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:15.123396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:15.123416 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:15.123813 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:15.124179 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:15.124184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:15.124187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:15.124366 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:15.124880 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:15.124888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:15.125210 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.125475 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.126025 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260927 22:18:15.128870 6521 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:15.129181 6521 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:15.129879 6521 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:15.131485 6204 ts_manager.cc:194] Registered new tserver with Master: f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331)
I20260927 22:18:15.132817 6204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:34595
W20260927 22:18:15.434176 6524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:15.434455 6524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:15.434548 6524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:15.441507 6524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:15.441607 6524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:15.441715 6524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:18:15.451227 6524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:43393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=38069
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:15.452436 6524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:15.453634 6524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:15.466859 6529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:15.466903 6532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:15.467057 6530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:15.468102 6524 server_base.cc:1034] running on GCE node
I20260927 22:18:15.468673 6524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:15.469620 6524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:15.470839 6524 hybrid_clock.cc:648] HybridClock initialized: now 1790547495470806 us; error 50 us; skew 500 ppm
I20260927 22:18:15.471145 6524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:15 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:15.474323 6524 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:18:15.475677 6524 webserver.cc:460] Webserver started at http://127.23.173.67:38069/ using document root <none> and password file <none>
I20260927 22:18:15.476331 6524 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:15.476488 6524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:15.481585 6524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:15.484191 6539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.485416 6524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:18:15.485590 6524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "d242fff3b64c42178e19230ae5e379f4"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "e8a95114ed2846a9a4e2d1ab1247c679"
server_key_iv: "4d150ab826c7e6022a5eb92634611dda"
server_key_version: "encryptionkey@0"
I20260927 22:18:15.486603 6524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:15.501267 6524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:15.505460 6524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:15.505729 6524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:15.506750 6524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:15.508014 6524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:15.508137 6524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.508279 6524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:15.508366 6524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:15.541857 6524 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43393
I20260927 22:18:15.541918 6651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43393 every 8 connection(s)
I20260927 22:18:15.543344 6524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:18:15.553336 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6524
Sep 27 22:18:15 dist-test-slave-3wc3 krb5kdc[5620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:15.563462 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.546020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:57341 (local address 127.23.173.126:37429)
0927 22:18:15.546244 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.546250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.547423 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 22:18:15.549145 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.549149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.549215 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.549302 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:15.550780 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.551503 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.553004 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.553389 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.553535 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.557539 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:15.557544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:15.557547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:15.557584 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:15.560391 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.561101 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.561113 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.561116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.561235 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.561673 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.561677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.561680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.561945 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:15.562158 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.562839 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.563086 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":13184}
I20260927 22:18:15.564379 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.546571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:57341)
0927 22:18:15.547238 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.547294 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.548901 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.549511 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.549520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.550067 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:15.550630 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.550646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.551624 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.551628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.552836 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.552847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.553627 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.553635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.553869 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.554719 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:15.554758 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:15.557277 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:15.560537 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:15.560545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:15.560561 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:15.560973 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:15.561346 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:15.561351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:15.561355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:15.561547 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:15.562128 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:15.562136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:15.562464 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.562738 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.563248 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260927 22:18:15.565933 6652 heartbeater.cc:347] Connected to a master server at 127.23.173.126:37429
I20260927 22:18:15.566295 6652 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:15.567175 6652 heartbeater.cc:511] Master 127.23.173.126:37429 requested a full tablet report, sending...
I20260927 22:18:15.568852 6204 ts_manager.cc:194] Registered new tserver with Master: d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393)
I20260927 22:18:15.568883 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.554318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.23.173.126:37429)
0927 22:18:15.555575 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.555585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.555599 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:18:15.555637 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.555640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.555699 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.555810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:15.557445 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.558326 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.559664 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.560146 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.560283 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.562971 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:15.562976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:15.562979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:15.563011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:15.566160 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.566846 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.566855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.566858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.567007 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.567395 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.567400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.567402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.567726 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:15.568019 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.568310 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.568539 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":1020,"threads_started":1}
I20260927 22:18:15.569725 6204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:57341
I20260927 22:18:15.571305 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:18:15.577819 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.572157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.23.173.126:37429)
0927 22:18:15.572338 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.572345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.572708 (+ 363us) server_negotiation.cc:407] Connection header received
0927 22:18:15.572954 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.572958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.573075 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.573215 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:15.573635 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.574409 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.575825 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.576230 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.576362 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.576847 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:15.577274 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:18:15.577420 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.577515 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.577586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:18:15.578637 6204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:18:15.585716 6204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:18:15.609067 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.597192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34331 (local address 127.0.0.1:33344)
0927 22:18:15.597901 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.597945 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.598254 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.600697 (+ 2443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.600708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.600748 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:15.601239 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.601254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.603821 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.603826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.606390 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.606424 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.608110 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.608116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.608301 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.608359 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.608498 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.608716 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":333,"threads_started":1}
I20260927 22:18:15.610085 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.597577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:43393 (local address 127.0.0.1:55956)
0927 22:18:15.599085 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.599101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.599225 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.602922 (+ 3697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.602926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.602948 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:15.603289 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.603303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.605106 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.605111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.607219 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.607233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.609240 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.609247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.609437 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.609466 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.609559 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.609683 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"mutex_wait_us":2,"thread_start_us":4769,"threads_started":1}
I20260927 22:18:15.610221 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.598157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.23.173.67:43393)
0927 22:18:15.600330 (+ 2173us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.600342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.600382 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:18:15.600508 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.600519 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.600793 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.602934 (+ 2141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:15.603497 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.604958 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.607403 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.608761 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.608882 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.609075 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.609621 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.609859 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1849,"threads_started":1}
I20260927 22:18:15.610481 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.597493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.23.173.66:34331)
0927 22:18:15.599880 (+ 2387us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.599892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.599932 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:18:15.600046 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.600055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.600418 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.601531 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:15.601888 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.603674 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.606586 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.607900 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.608050 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.608286 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.608539 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.608756 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2186,"thread_start_us":2004,"threads_started":1}
I20260927 22:18:15.611752 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.598350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40439 (local address 127.0.0.1:59276)
0927 22:18:15.602645 (+ 4295us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.602658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.602756 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.604162 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.604167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.604191 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:15.604669 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.604681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.606668 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.606673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.608533 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.608547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.610920 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.610926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.611095 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.611129 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.611236 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.611363 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4179,"mutex_wait_us":1,"thread_start_us":643,"threads_started":1}
I20260927 22:18:15.611754 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.598844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.23.173.65:40439)
0927 22:18:15.603592 (+ 4748us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.603600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.603627 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:18:15.603722 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.603732 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.603964 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.604152 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:15.604908 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.606492 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.608714 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.610635 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.610863 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.611051 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.611279 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.611430 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4579,"thread_start_us":4436,"threads_started":1}
I20260927 22:18:15.614207 6456 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 (DEFAULT_TABLE table=test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:15.614208 6587 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 (DEFAULT_TABLE table=test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:15.615128 6324 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 (DEFAULT_TABLE table=test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:15.615674 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4abb24c9c648ea87eb178cea426c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:15.615732 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4abb24c9c648ea87eb178cea426c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:15.616222 6324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4abb24c9c648ea87eb178cea426c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:15.628762 6668 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Bootstrap starting.
I20260927 22:18:15.632066 6668 tablet_bootstrap.cc:654] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:15.633080 6668 log.cc:824] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:15.634100 6669 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Bootstrap starting.
I20260927 22:18:15.635020 6670 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Bootstrap starting.
I20260927 22:18:15.638284 6669 tablet_bootstrap.cc:654] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:15.639238 6669 log.cc:824] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:15.639307 6670 tablet_bootstrap.cc:654] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:15.640543 6670 log.cc:824] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:15.642249 6669 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: No bootstrap required, opened a new log
I20260927 22:18:15.642309 6668 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: No bootstrap required, opened a new log
I20260927 22:18:15.642534 6669 ts_tablet_manager.cc:1397] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:18:15.642578 6668 ts_tablet_manager.cc:1397] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260927 22:18:15.643483 6670 tablet_bootstrap.cc:492] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: No bootstrap required, opened a new log
I20260927 22:18:15.643739 6670 ts_tablet_manager.cc:1397] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:18:15.650805 6668 raft_consensus.cc:348] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.651237 6668 raft_consensus.cc:374] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:15.651345 6668 raft_consensus.cc:729] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767caf9a07da46e09bd21cfddd451a30, State: Initialized, Role: FOLLOWER
I20260927 22:18:15.652002 6668 consensus_queue.cc:260] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.653451 6669 raft_consensus.cc:348] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.653855 6669 raft_consensus.cc:374] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:15.653944 6669 raft_consensus.cc:729] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d242fff3b64c42178e19230ae5e379f4, State: Initialized, Role: FOLLOWER
I20260927 22:18:15.654435 6670 raft_consensus.cc:348] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.654834 6670 raft_consensus.cc:374] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:15.654929 6670 raft_consensus.cc:729] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04703c4d11d4055aedadc66736bfc88, State: Initialized, Role: FOLLOWER
I20260927 22:18:15.654780 6669 consensus_queue.cc:260] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.655678 6670 consensus_queue.cc:260] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.659543 6652 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:15.660184 6669 ts_tablet_manager.cc:1428] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260927 22:18:15.660743 6521 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:15.661346 6668 ts_tablet_manager.cc:1428] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260927 22:18:15.661728 6389 heartbeater.cc:503] Master 127.23.173.126:37429 was elected leader, sending a full tablet report...
I20260927 22:18:15.662526 6670 ts_tablet_manager.cc:1428] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
W20260927 22:18:15.794948 6653 tablet.cc:2369] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:15.797873 6676 raft_consensus.cc:482] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:18:15.798211 6676 raft_consensus.cc:504] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.799862 6676 leader_election.cc:288] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439), d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393)
I20260927 22:18:15.806316 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.800383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:33611 (local address 127.23.173.67:43393)
0927 22:18:15.800594 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.800600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.801148 (+ 548us) server_negotiation.cc:407] Connection header received
0927 22:18:15.801382 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.801387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.801455 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.801557 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:15.802114 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.802845 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.804620 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.805385 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.805563 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.805642 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.805912 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.806061 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:18:15.806485 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.800249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:43393 (local address 127.23.173.66:33611)
0927 22:18:15.801033 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.801048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.801229 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.801572 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.801577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.801603 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:15.801932 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.801941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.802980 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.802988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.804470 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.804482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.805604 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.805611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.805749 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.805771 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.805875 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.806032 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":20,"thread_start_us":501,"threads_started":1}
I20260927 22:18:15.806936 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.800435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:34973 (local address 127.23.173.65:40439)
0927 22:18:15.800618 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.800625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.801664 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 22:18:15.801883 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.801890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.801999 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.802105 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:15.802652 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.803370 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.805111 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.805881 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.806121 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.806198 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.806569 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.806724 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:18:15.807519 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.800463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40439 (local address 127.23.173.66:34973)
0927 22:18:15.801547 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:15.801561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:15.801726 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:15.802123 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:15.802127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:15.802150 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:15.802497 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.802506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.803544 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.803549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.804951 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:15.804963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.806167 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.806174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:15.806348 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.806369 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:18:15.807133 (+ 764us) client_negotiation.cc:240] Negotiation successful
0927 22:18:15.807250 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":1,"thread_start_us":728,"threads_started":1}
I20260927 22:18:15.807489 6607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4abb24c9c648ea87eb178cea426c34" candidate_uuid: "f04703c4d11d4055aedadc66736bfc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d242fff3b64c42178e19230ae5e379f4" is_pre_election: true
I20260927 22:18:15.808053 6607 raft_consensus.cc:2454] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04703c4d11d4055aedadc66736bfc88 in term 0.
I20260927 22:18:15.808758 6411 leader_election.cc:302] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d242fff3b64c42178e19230ae5e379f4, f04703c4d11d4055aedadc66736bfc88; no voters:
I20260927 22:18:15.808672 6344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4abb24c9c648ea87eb178cea426c34" candidate_uuid: "f04703c4d11d4055aedadc66736bfc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767caf9a07da46e09bd21cfddd451a30" is_pre_election: true
I20260927 22:18:15.809135 6676 raft_consensus.cc:2781] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:18:15.809178 6344 raft_consensus.cc:2454] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f04703c4d11d4055aedadc66736bfc88 in term 0.
I20260927 22:18:15.809273 6676 raft_consensus.cc:482] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:18:15.809417 6676 raft_consensus.cc:3037] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:15.811849 6676 raft_consensus.cc:504] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.812486 6676 leader_election.cc:288] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [CANDIDATE]: Term 1 election: Requested vote from peers 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439), d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393)
I20260927 22:18:15.812844 6344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4abb24c9c648ea87eb178cea426c34" candidate_uuid: "f04703c4d11d4055aedadc66736bfc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767caf9a07da46e09bd21cfddd451a30"
I20260927 22:18:15.812870 6607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4abb24c9c648ea87eb178cea426c34" candidate_uuid: "f04703c4d11d4055aedadc66736bfc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d242fff3b64c42178e19230ae5e379f4"
I20260927 22:18:15.813026 6344 raft_consensus.cc:3037] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:15.813087 6607 raft_consensus.cc:3037] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:15.815356 6344 raft_consensus.cc:2454] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04703c4d11d4055aedadc66736bfc88 in term 1.
I20260927 22:18:15.815397 6607 raft_consensus.cc:2454] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f04703c4d11d4055aedadc66736bfc88 in term 1.
I20260927 22:18:15.815757 6410 leader_election.cc:302] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 767caf9a07da46e09bd21cfddd451a30, f04703c4d11d4055aedadc66736bfc88; no voters:
I20260927 22:18:15.816071 6676 raft_consensus.cc:2781] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:15.816524 6676 raft_consensus.cc:686] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 1 LEADER]: Becoming Leader. State: Replica: f04703c4d11d4055aedadc66736bfc88, State: Running, Role: LEADER
I20260927 22:18:15.816978 6676 consensus_queue.cc:237] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } }
I20260927 22:18:15.821668 6204 catalog_manager.cc:5495] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 reported cstate change: term changed from 0 to 1, leader changed from <none> to f04703c4d11d4055aedadc66736bfc88 (127.23.173.66). New cstate: current_term: 1 leader_uuid: "f04703c4d11d4055aedadc66736bfc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04703c4d11d4055aedadc66736bfc88" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 } health_report { overall_health: UNKNOWN } } }
I20260927 22:18:15.844202 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.838758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.23.173.66:34331)
0927 22:18:15.838978 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.838983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.839003 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:18:15.839233 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.839237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.839317 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.839479 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:15.839841 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.840553 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.842093 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.842518 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.842654 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.843141 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:15.843627 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:18:15.843798 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.843855 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.843941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:18:15.852377 6607 raft_consensus.cc:1261] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Refusing update from remote peer f04703c4d11d4055aedadc66736bfc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:15.852398 6344 raft_consensus.cc:1261] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Refusing update from remote peer f04703c4d11d4055aedadc66736bfc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:15.853323 6679 consensus_queue.cc:1035] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d242fff3b64c42178e19230ae5e379f4" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:15.853672 6676 consensus_queue.cc:1035] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767caf9a07da46e09bd21cfddd451a30" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:18:15.858060 6522 tablet.cc:2369] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:15.871439 6685 mvcc.cc:205] Tried to move back new op lower bound from 7334082543001763840 to 7334082542870835200. Current Snapshot: MvccSnapshot[applied={T|T < 7334082543001763840}]
I20260927 22:18:15.879103 6202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:15.879431 6202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:15.883201 6202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Sending DeleteTablet for 3 replicas of tablet 4c4abb24c9c648ea87eb178cea426c34
I20260927 22:18:15.884289 6456 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:15 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260927 22:18:15.884423 6324 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:15 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:59276
I20260927 22:18:15.884574 6587 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4abb24c9c648ea87eb178cea426c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:15 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:55956
I20260927 22:18:15.885545 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6241
I20260927 22:18:15.887127 6692 tablet_replica.cc:330] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: stopping tablet replica
I20260927 22:18:15.887579 6693 tablet_replica.cc:330] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: stopping tablet replica
I20260927 22:18:15.888051 6693 raft_consensus.cc:2229] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:15.888648 6693 raft_consensus.cc:2258] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:15.889132 6692 raft_consensus.cc:2229] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:15.889683 6692 raft_consensus.cc:2258] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:15.894518 6692 ts_tablet_manager.cc:1905] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:15.898788 6691 tablet_replica.cc:330] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: stopping tablet replica
I20260927 22:18:15.899272 6691 raft_consensus.cc:2229] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:18:15.899772 6693 ts_tablet_manager.cc:1905] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:15.900322 6691 raft_consensus.cc:2258] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:15.903116 6691 ts_tablet_manager.cc:1905] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:15.904229 6692 ts_tablet_manager.cc:1918] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:15.904378 6692 log.cc:1196] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/4c4abb24c9c648ea87eb178cea426c34
I20260927 22:18:15.904686 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:15.887144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.23.173.65:40439)
0927 22:18:15.887324 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:15.887339 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:15.887359 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:18:15.887414 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:15.887419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:15.887510 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:15.887623 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:15.889403 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.890871 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.892910 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:15.893765 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:15.893921 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:15.897329 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:15.897347 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:15.897369 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:15.897422 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:15.901440 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.902246 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.902254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.902258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.902411 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:15.902855 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:15.902861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:15.902865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:15.903320 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:15.903647 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:15.903986 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:18:15.904249 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:18:15.904982 6692 ts_tablet_manager.cc:1939] T 4c4abb24c9c648ea87eb178cea426c34 P 767caf9a07da46e09bd21cfddd451a30: Deleting consensus metadata
I20260927 22:18:15.905359 6384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:15.906909 6693 ts_tablet_manager.cc:1918] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:15.907066 6693 log.cc:1196] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/4c4abb24c9c648ea87eb178cea426c34
I20260927 22:18:15.907608 6693 ts_tablet_manager.cc:1939] T 4c4abb24c9c648ea87eb178cea426c34 P d242fff3b64c42178e19230ae5e379f4: Deleting consensus metadata
I20260927 22:18:15.909392 6190 catalog_manager.cc:4841] TS d242fff3b64c42178e19230ae5e379f4 (127.23.173.67:43393): tablet 4c4abb24c9c648ea87eb178cea426c34 (table test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]) successfully deleted
I20260927 22:18:15.912472 6691 ts_tablet_manager.cc:1918] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:15.912771 6691 log.cc:1196] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/4c4abb24c9c648ea87eb178cea426c34
I20260927 22:18:15.913406 6691 ts_tablet_manager.cc:1939] T 4c4abb24c9c648ea87eb178cea426c34 P f04703c4d11d4055aedadc66736bfc88: Deleting consensus metadata
I20260927 22:18:15.915066 6189 catalog_manager.cc:4841] TS f04703c4d11d4055aedadc66736bfc88 (127.23.173.66:34331): tablet 4c4abb24c9c648ea87eb178cea426c34 (table test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]) successfully deleted
I20260927 22:18:15.999936 6189 catalog_manager.cc:4841] TS 767caf9a07da46e09bd21cfddd451a30 (127.23.173.65:40439): tablet 4c4abb24c9c648ea87eb178cea426c34 (table test-table [id=bcaa47a0864e493f8bcfebb5b03508e1]) successfully deleted
I20260927 22:18:16.000532 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6241
I20260927 22:18:16.024223 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6393
I20260927 22:18:16.049381 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:16.025039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.23.173.66:34331)
0927 22:18:16.025204 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:16.025208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:16.025262 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:18:16.025467 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:16.025470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:16.025577 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:16.025657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:16.027484 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:16.028232 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:16.030286 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:16.030701 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:16.030789 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:16.043025 (+ 12236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:16.043039 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:16.043054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:16.043088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:16.046561 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:16.047424 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:16.047434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:16.047436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:16.047567 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:16.047969 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:16.047972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:16.047974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:16.048332 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:16.048562 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:16.048910 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:18:16.049066 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:18:16.050122 6516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:16.170499 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6393
I20260927 22:18:16.191404 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6524
I20260927 22:18:16.203967 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:16.192131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.23.173.67:43393)
0927 22:18:16.192320 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:16.192324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:16.192341 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:18:16.192492 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:16.192497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:16.192567 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:16.192650 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:16.193915 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:16.194693 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:16.195906 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:16.196263 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:16.196350 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:16.198909 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:16.198921 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:16.198937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:16.198982 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:16.201549 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:16.202229 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:16.202236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:16.202239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:16.202362 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:16.202686 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:16.202689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:16.202691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:16.203026 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:16.203246 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:16.203450 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:18:16.203603 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:18:16.204638 6647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:16.293318 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6524
I20260927 22:18:16.310335 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6171
I20260927 22:18:16.311532 6234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:16.426007 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6171
I20260927 22:18:16.449470 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--webserver_interface=127.23.173.126
--webserver_port=38615
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:16.773838 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:16.774240 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:16.774413 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:16.782217 6698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:16.782351 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:16.782455 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:16.782553 6698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:16.782639 6698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:16.792601 6698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=38615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:16.794023 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:16.795356 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:16.808133 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:16.808210 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:16.808185 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:16.815701 6698 server_base.cc:1034] running on GCE node
I20260927 22:18:16.816555 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:16.818084 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:16.819317 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790547496819280 us; error 54 us; skew 500 ppm
I20260927 22:18:16.819670 6698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:16 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:16.822934 6698 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:16.824482 6698 webserver.cc:460] Webserver started at http://127.23.173.126:38615/ using document root <none> and password file <none>
I20260927 22:18:16.825186 6698 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:16.825316 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:16.830698 6698 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:16.833438 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:16.834736 6698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:16.834921 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "294b88e0d6f947e19948909a0e07b11a"
format_stamp: "Formatted at 2026-09-27 22:18:10 on dist-test-slave-3wc3"
server_key: "8c2affedd2a8f9d7959f62f0b451289d"
server_key_iv: "9e81d888405c36b4f5f9a5cb4cb0d1f7"
server_key_version: "encryptionkey@0"
I20260927 22:18:16.835839 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:16.856837 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:16.860857 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:16.861135 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:16.880033 6698 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37429
I20260927 22:18:16.880052 6764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37429 every 8 connection(s)
I20260927 22:18:16.881552 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:16.882567 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6698
I20260927 22:18:16.883419 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:40439
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=36659
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:16.888753 6765 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:18:16.892067 6765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap starting.
I20260927 22:18:16.912384 6765 log.cc:824] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:16.928036 6765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:18:16.928597 6765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a: Bootstrap complete.
I20260927 22:18:16.937665 6765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:16.938560 6765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Initialized, Role: FOLLOWER
I20260927 22:18:16.939162 6765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:16.939393 6765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:16.939555 6765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:16.939739 6765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:18:16.943753 6765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:16.944237 6765 leader_election.cc:302] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 294b88e0d6f947e19948909a0e07b11a; no voters:
I20260927 22:18:16.944913 6765 leader_election.cc:288] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:18:16.945050 6769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:18:16.946293 6769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [term 3 LEADER]: Becoming Leader. State: Replica: 294b88e0d6f947e19948909a0e07b11a, State: Running, Role: LEADER
I20260927 22:18:16.946867 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } }
I20260927 22:18:16.947155 6765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:16.949262 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 294b88e0d6f947e19948909a0e07b11a. Latest consensus state: current_term: 3 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:16.949719 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:16.949443 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "294b88e0d6f947e19948909a0e07b11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294b88e0d6f947e19948909a0e07b11a" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37429 } } }
I20260927 22:18:16.949841 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294b88e0d6f947e19948909a0e07b11a [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:16.963006 6776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:16.971342 6776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:16.972739 6776 catalog_manager.cc:1259] Loaded cluster ID: 28ea3f8fbd6e4f9a8ef400f24872ec1f
I20260927 22:18:16.972841 6776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:18:16.974951 6776 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7eff15ad59ad google::LogMessage::Fail() at ??:0
@ 0x7eff15ad888a google::LogMessage::SendToLog() at ??:0
@ 0x7eff15ad54de google::LogMessage::Flush() at ??:0
@ 0x7eff15ad6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7eff2b50c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7eff2b509976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7eff2b591610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7eff2b591439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff18721345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7eff1872491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7eff18724799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff186ffeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7eff1b9d96db start_thread at ??:0
@ 0x7eff1496f61f clone at ??:0
W20260927 22:18:17.241466 6767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:17.241776 6767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:17.242024 6767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:17.249465 6767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:17.249616 6767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:17.249792 6767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:18:17.259735 6767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:40439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=36659
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:17.261003 6767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:17.262406 6767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:17.275228 6806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:17.275157 6809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:17.275189 6807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:17.275928 6767 server_base.cc:1034] running on GCE node
I20260927 22:18:17.276546 6767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:17.277483 6767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:17.278699 6767 hybrid_clock.cc:648] HybridClock initialized: now 1790547497278644 us; error 69 us; skew 500 ppm
I20260927 22:18:17.278999 6767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:17 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:17.281939 6767 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:18:17.283336 6767 webserver.cc:460] Webserver started at http://127.23.173.65:36659/ using document root <none> and password file <none>
I20260927 22:18:17.283990 6767 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:17.284133 6767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:17.289201 6767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:17.291791 6816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:17.293032 6767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:18:17.293207 6767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "767caf9a07da46e09bd21cfddd451a30"
format_stamp: "Formatted at 2026-09-27 22:18:11 on dist-test-slave-3wc3"
server_key: "866339fa80e7c3dc510414c10e3bff29"
server_key_iv: "7fe6e102a71737a030fad6bbcf597128"
server_key_version: "encryptionkey@0"
I20260927 22:18:17.294203 6767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:17.306802 6767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:17.310817 6767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:17.311098 6767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:17.311982 6767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:17.313220 6767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:17.313350 6767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:17.313478 6767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:17.313565 6767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:17.345820 6767 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40439
I20260927 22:18:17.345839 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40439 every 8 connection(s)
I20260927 22:18:17.347309 6767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:18:17.352190 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.350437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:60635)
0927 22:18:17.351161 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.351373 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":269,"threads_started":1}
W20260927 22:18:17.352846 6929 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37429 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:37429: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
I20260927 22:18:17.353621 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.353311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:42127)
0927 22:18:17.353451 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.353524 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:18:17.354445 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.354139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:48591)
0927 22:18:17.354268 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.354345 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:18:17.354669 6929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:18:17.356339 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6767
I20260927 22:18:17.356956 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:34331
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=41873
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:17.686848 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:17.687129 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:17.687309 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:17.694530 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:17.694636 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:17.694746 6932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:18:17.704627 6932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:34331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=41873
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:17.705929 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:17.707224 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:17.720368 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:17.720332 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:17.720674 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:17.721212 6932 server_base.cc:1034] running on GCE node
I20260927 22:18:17.721870 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:17.722851 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:17.724079 6932 hybrid_clock.cc:648] HybridClock initialized: now 1790547497724034 us; error 40 us; skew 500 ppm
I20260927 22:18:17.724398 6932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:17 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:17.727484 6932 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:18:17.728971 6932 webserver.cc:460] Webserver started at http://127.23.173.66:41873/ using document root <none> and password file <none>
I20260927 22:18:17.729658 6932 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:17.729806 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:17.735080 6932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:18:17.737689 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:17.739185 6932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:17.739401 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "f04703c4d11d4055aedadc66736bfc88"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "88b5a2e2e1c502ed83710dcbbbeead6c"
server_key_iv: "a0a1b3bfa5f8cbe30c58a94ccd021db8"
server_key_version: "encryptionkey@0"
I20260927 22:18:17.740403 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:17.766815 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:17.770963 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:17.771260 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:17.772269 6932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:17.773649 6932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:17.773769 6932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:17.773942 6932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:17.774047 6932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:18:17.807469 6932 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34331
I20260927 22:18:17.807564 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34331 every 8 connection(s)
I20260927 22:18:17.808899 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:18:17.809774 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 6932
I20260927 22:18:17.810613 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:43393
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:37429
--webserver_port=38069
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:17.815945 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.812469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:49069)
0927 22:18:17.814711 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.814998 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1956,"thread_start_us":1781,"threads_started":1}
W20260927 22:18:17.816622 7060 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37429 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:37429: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
I20260927 22:18:17.817472 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.817132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:40677)
0927 22:18:17.817294 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.817371 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:18:17.818332 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:17.818016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:42043)
0927 22:18:17.818145 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:17.818227 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 22:18:17.818631 7060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:18:18.133708 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:18.134044 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:18.134214 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:18.141346 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:18.141516 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:18.141741 7063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:18:18.151800 7063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:43393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=38069
--tserver_master_addrs=127.23.173.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:18.153084 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:18.154451 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:18.167366 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:18.167490 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:18.167629 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:18.168579 7063 server_base.cc:1034] running on GCE node
I20260927 22:18:18.169126 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:18.170127 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:18.171342 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790547498171299 us; error 30 us; skew 500 ppm
I20260927 22:18:18.171615 7063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:18 dist-test-slave-3wc3 krb5kdc[5620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547498, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:18.174949 7063 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:18:18.176403 7063 webserver.cc:460] Webserver started at http://127.23.173.67:38069/ using document root <none> and password file <none>
I20260927 22:18:18.177062 7063 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:18.177196 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:18.182399 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:18:18.184993 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:18.186316 7063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:18.186489 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "d242fff3b64c42178e19230ae5e379f4"
format_stamp: "Formatted at 2026-09-27 22:18:12 on dist-test-slave-3wc3"
server_key: "e8a95114ed2846a9a4e2d1ab1247c679"
server_key_iv: "4d150ab826c7e6022a5eb92634611dda"
server_key_version: "encryptionkey@0"
I20260927 22:18:18.187481 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:18.207967 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:18.211745 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:18.212009 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:18.212962 7063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:18.214200 7063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:18.214339 7063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:18.214478 7063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:18.214566 7063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:18.246968 7063 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43393
I20260927 22:18:18.247049 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43393 every 8 connection(s)
I20260927 22:18:18.248416 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:18:18.253057 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:18.251287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:56019)
0927 22:18:18.251985 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:18.252208 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
I20260927 22:18:18.253409 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7063
W20260927 22:18:18.253765 7191 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:37429 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:37429: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
I20260927 22:18:18.254603 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:18.254286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:35661)
0927 22:18:18.254419 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:18.254482 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:18:18.255571 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:18.255226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:43859)
0927 22:18:18.255387 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:18.255465 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:18:18.255805 7191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:18:18.355922 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:18.355149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:46165)
0927 22:18:18.355569 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:18.355691 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":201,"threads_started":1}
I20260927 22:18:18.819852 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:18.819120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:52157)
0927 22:18:18.819535 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:18.819643 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":202,"threads_started":1}
I20260927 22:18:19.256973 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:19.256236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:50605)
0927 22:18:19.256642 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:19.256747 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260927 22:18:19.357579 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:19.356933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:53515)
0927 22:18:19.357352 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:19.357428 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260927 22:18:19.821411 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:19.820711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:46647)
0927 22:18:19.821177 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:19.821255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260927 22:18:20.258665 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:20.257942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:42013)
0927 22:18:20.258440 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:20.258513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":234,"threads_started":1}
I20260927 22:18:20.359130 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:20.358445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:41885)
0927 22:18:20.358885 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:20.358967 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260927 22:18:20.823190 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:20.822277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:50875)
0927 22:18:20.822818 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:20.822955 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":311,"threads_started":1}
I20260927 22:18:21.260079 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:21.259455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:48809)
0927 22:18:21.259839 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:21.259915 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":208,"threads_started":1}
I20260927 22:18:21.360733 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:21.360032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:37635)
0927 22:18:21.360468 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:21.360556 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260927 22:18:21.824755 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:21.824022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:36541)
0927 22:18:21.824509 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:21.824591 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260927 22:18:22.261837 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:22.261056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:60909)
0927 22:18:22.261528 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:22.261604 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":260,"threads_started":1}
I20260927 22:18:22.362347 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:22.361577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:55317)
0927 22:18:22.362110 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:22.362188 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260927 22:18:22.826362 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:22.825606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:60041)
0927 22:18:22.826115 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:22.826194 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":248,"threads_started":1}
I20260927 22:18:23.263381 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:23.262685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:54475)
0927 22:18:23.263148 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:23.263223 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260927 22:18:23.363842 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:23.363174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:40555)
0927 22:18:23.363599 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:23.363687 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260927 22:18:23.827816 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:23.827175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:34145)
0927 22:18:23.827579 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:23.827665 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260927 22:18:24.264832 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:24.264167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:42367)
0927 22:18:24.264592 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:24.264679 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":218,"threads_started":1}
I20260927 22:18:24.365476 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:24.364733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:45371)
0927 22:18:24.365191 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:24.365269 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":216,"threads_started":1}
I20260927 22:18:24.829493 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:24.828768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:45781)
0927 22:18:24.829224 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:24.829299 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":215,"threads_started":1}
I20260927 22:18:25.266693 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:25.265657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:46621)
0927 22:18:25.266277 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:25.266377 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260927 22:18:25.367020 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:25.366291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:54057)
0927 22:18:25.366773 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:25.366861 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
I20260927 22:18:25.831300 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:25.830428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:52231)
0927 22:18:25.830972 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:25.831091 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260927 22:18:26.268268 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:26.267642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:37303)
0927 22:18:26.268045 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:26.268121 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":209,"threads_started":1}
I20260927 22:18:26.368624 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:26.367892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:58327)
0927 22:18:26.368381 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:26.368459 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":239,"threads_started":1}
I20260927 22:18:26.833060 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:26.832339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:52709)
0927 22:18:26.832805 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:26.832887 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260927 22:18:27.270156 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:27.269325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:58791)
0927 22:18:27.269797 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:27.269875 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260927 22:18:27.370386 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:27.369659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:45565)
0927 22:18:27.370153 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:27.370233 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":226,"threads_started":1}
I20260927 22:18:27.834726 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:27.834053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:52803)
0927 22:18:27.834487 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:27.834571 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260927 22:18:28.271622 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:28.270977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:36169)
0927 22:18:28.271396 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:28.271472 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260927 22:18:28.371992 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:28.371292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:52255)
0927 22:18:28.371723 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:28.371815 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260927 22:18:28.836457 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:28.835747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:58965)
0927 22:18:28.836192 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:28.836290 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260927 22:18:29.273136 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:29.272473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:56431)
0927 22:18:29.272860 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:29.272951 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":195,"threads_started":1}
I20260927 22:18:29.373823 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:29.372951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:35181)
0927 22:18:29.373463 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:29.373581 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260927 22:18:29.838193 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:29.837465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:42765)
0927 22:18:29.837893 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:29.838041 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260927 22:18:30.274794 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:30.274137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:57337)
0927 22:18:30.274556 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:30.274638 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":206,"threads_started":1}
I20260927 22:18:30.375492 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:30.374732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:42937)
0927 22:18:30.375218 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:30.375307 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":243,"threads_started":1}
I20260927 22:18:30.839766 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:30.839017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:53355)
0927 22:18:30.839493 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:30.839582 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
I20260927 22:18:31.276463 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:31.275736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:38365)
0927 22:18:31.276191 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:31.276278 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":216,"threads_started":1}
I20260927 22:18:31.377141 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:31.376447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:52791)
0927 22:18:31.376868 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:31.376949 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260927 22:18:31.841650 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:31.840787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:55337)
0927 22:18:31.841271 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:31.841407 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":243,"threads_started":1}
I20260927 22:18:32.278014 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:32.277309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:34589)
0927 22:18:32.277722 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:32.277813 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":210,"threads_started":1}
I20260927 22:18:32.379196 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:32.378357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.65:32847)
0927 22:18:32.378816 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:32.378937 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":244,"threads_started":1}
I20260927 22:18:32.843466 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:32.842733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.66:56539)
0927 22:18:32.843188 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:32.843297 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
I20260927 22:18:33.255426 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6767
I20260927 22:18:33.274282 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:33.256922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.23.173.65:40439)
0927 22:18:33.257567 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:33.257592 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:33.257647 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:18:33.259687 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:33.259696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:33.260146 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:33.260495 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:33.261816 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.263215 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.264431 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.265140 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.265275 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:33.267912 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:33.267920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:33.267935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:33.267971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:33.271210 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.271805 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.271811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.271818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.272020 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.272370 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.272373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.272375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.272888 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:33.273116 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:33.273489 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 22:18:33.273920 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
I20260927 22:18:33.275995 6924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:33.280067 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:33.279225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37429 (local address 127.23.173.67:60295)
0927 22:18:33.279703 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:33.279836 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:37429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":268,"threads_started":1}
I20260927 22:18:33.360530 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6767
I20260927 22:18:33.378410 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 6932
I20260927 22:18:33.395840 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:33.379387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.23.173.66:34331)
0927 22:18:33.379883 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:33.379898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:33.379934 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:18:33.381407 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:33.381416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:33.381849 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:33.382239 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:33.383746 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.384977 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.386293 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.386966 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.387100 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:33.389670 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:33.389677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:33.389693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:33.389728 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:33.392967 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.393581 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.393586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.393593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.393747 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.394125 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.394128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.394130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.394629 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:33.394845 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:33.395155 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:18:33.395569 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":234,"threads_started":1}
I20260927 22:18:33.397253 7055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:33.482954 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 6932
I20260927 22:18:33.500161 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7063
I20260927 22:18:33.518359 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:33.501175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.23.173.67:43393)
0927 22:18:33.501766 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:33.501780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:33.501820 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:18:33.503477 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:33.503486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:33.503923 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:33.504281 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:33.505763 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.507055 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.508348 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:33.509013 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:33.509218 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:33.511998 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:33.512005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:33.512019 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:33.512059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:33.514890 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.515622 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.515628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.515634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.515790 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:33.516185 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:33.516191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:33.516194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:33.516688 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:33.517006 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:33.517429 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:18:33.517834 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
I20260927 22:18:33.519654 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:33.604653 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7063
2026-09-27T22:18:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24370 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:18:33 dist-test-slave-3wc3 krb5kdc[7248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:18:33 dist-test-slave-3wc3 krb5kdc[7248](info): set up 2 sockets
Sep 27 22:18:33 dist-test-slave-3wc3 krb5kdc[7248](info): commencing operation
krb5kdc: starting...
W20260927 22:18:34.702584 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:18:34 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:18:34.939536 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:35.259570 7261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:35.259837 7261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:35.259925 7261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:35.266942 7261 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:18:35.267040 7261 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:18:35.267077 7261 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:18:35.267143 7261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:35.267187 7261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:35.267254 7261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:35.267299 7261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:35.267352 7261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:35.277093 7261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:35.278414 7261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:35.279706 7261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:35.291946 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:35.291997 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:35.291911 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:35.292443 7261 server_base.cc:1034] running on GCE node
I20260927 22:18:35.293224 7261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:35.294605 7261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:35.295847 7261 hybrid_clock.cc:648] HybridClock initialized: now 1790547515295794 us; error 37 us; skew 500 ppm
I20260927 22:18:35.296177 7261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:35 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547515, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:35.303102 7261 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:35.304507 7261 webserver.cc:460] Webserver started at http://127.23.173.126:45617/ using document root <none> and password file <none>
I20260927 22:18:35.305238 7261 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:35.305370 7261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:35.305687 7261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:35.308688 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c557d0b9625744e48c9ae3ad4fddf023"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "87a8d92ce4cd2d1d2b32f0c82d39e874"
server_key_iv: "b0717db577e7c8e03e232db468b88a2f"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.309473 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c557d0b9625744e48c9ae3ad4fddf023"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "87a8d92ce4cd2d1d2b32f0c82d39e874"
server_key_iv: "b0717db577e7c8e03e232db468b88a2f"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.314502 7261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:18:35.317791 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:35.319116 7261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:18:35.319312 7261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c557d0b9625744e48c9ae3ad4fddf023"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "87a8d92ce4cd2d1d2b32f0c82d39e874"
server_key_iv: "b0717db577e7c8e03e232db468b88a2f"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.319540 7261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:35.337074 7261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:35.341236 7261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:35.341535 7261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:35.361251 7261 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:41723
I20260927 22:18:35.361302 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:41723 every 8 connection(s)
I20260927 22:18:35.362702 7261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:35.366312 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:35.371742 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7261
I20260927 22:18:35.371968 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:18:35.372877 24245 external_mini_cluster.cc:1428] Setting key ad82f306cee707370118dae20713c25e
I20260927 22:18:35.377753 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap starting.
I20260927 22:18:35.380862 7328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:35.381878 7328 log.cc:824] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:35.384632 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: No bootstrap required, opened a new log
Sep 27 22:18:35 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:35.392333 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:35.392720 7328 raft_consensus.cc:374] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:35.392854 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Initialized, Role: FOLLOWER
I20260927 22:18:35.393426 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:35.393675 7328 raft_consensus.cc:388] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:35.393821 7328 raft_consensus.cc:482] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:35.394021 7328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:35.397291 7328 raft_consensus.cc:504] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:35.397893 7328 leader_election.cc:302] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c557d0b9625744e48c9ae3ad4fddf023; no voters:
I20260927 22:18:35.398329 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:35.375090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.23.173.126:41723)
0927 22:18:35.375963 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:35.375975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:35.376019 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:18:35.377785 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:35.377809 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:35.378316 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:35.378720 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:35.380480 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.382016 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.383954 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.384701 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.384939 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:35.389543 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:35.389554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:35.389573 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:35.389625 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:35.393331 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:35.394231 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:35.394240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:35.394247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:35.394484 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:35.394898 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:35.394903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:35.394907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:35.395675 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:35.395979 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:35.396451 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 22:18:35.396878 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":385,"threads_started":1}
I20260927 22:18:35.398592 7328 leader_election.cc:288] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:18:35.398767 7333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:35.401008 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:35.402242 7333 raft_consensus.cc:686] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 LEADER]: Becoming Leader. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Running, Role: LEADER
I20260927 22:18:35.402913 7333 consensus_queue.cc:237] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:35.405537 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c557d0b9625744e48c9ae3ad4fddf023. Latest consensus state: current_term: 1 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:18:35.405563 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:18:35.405992 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:35.406013 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:35.407563 7338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:35.412799 7338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:35.421029 7338 catalog_manager.cc:1347] Generated new cluster ID: 595a15ac28af47349ccd5a015f35eca3
I20260927 22:18:35.421149 7338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:35.444839 7338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:18:35.446350 7338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:35.458280 7338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Generated new TSK 0
I20260927 22:18:35.459131 7338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:18:35.524320 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:35.852136 7354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:35.852397 7354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:35.852551 7354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:35.859625 7354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:35.859740 7354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:35.859890 7354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:18:35.869262 7354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:35.870532 7354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:35.871838 7354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:35.885367 7359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:35.885411 7360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:35.885440 7362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:35.886154 7354 server_base.cc:1034] running on GCE node
I20260927 22:18:35.886831 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:35.887753 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:35.888981 7354 hybrid_clock.cc:648] HybridClock initialized: now 1790547515888934 us; error 33 us; skew 500 ppm
I20260927 22:18:35.889314 7354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:35 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547515, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:35.892488 7354 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:18:35.893922 7354 webserver.cc:460] Webserver started at http://127.23.173.65:45309/ using document root <none> and password file <none>
I20260927 22:18:35.894660 7354 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:35.894801 7354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:35.895105 7354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:35.898160 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "bd8a7e57d8344f7b937f818fccdab501"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "96e896248f813ea49afe3c752cce8a83"
server_key_iv: "5f0e245e51a700748a62b2d47bebf3a1"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.898892 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "bd8a7e57d8344f7b937f818fccdab501"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "96e896248f813ea49afe3c752cce8a83"
server_key_iv: "5f0e245e51a700748a62b2d47bebf3a1"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.904053 7354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:18:35.907392 7369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:35.908576 7354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 22:18:35.908742 7354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "bd8a7e57d8344f7b937f818fccdab501"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "96e896248f813ea49afe3c752cce8a83"
server_key_iv: "5f0e245e51a700748a62b2d47bebf3a1"
server_key_version: "encryptionkey@0"
I20260927 22:18:35.908944 7354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:35.922576 7354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:35.927066 7354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:35.927353 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:35.928294 7354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:35.929558 7354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:35.929659 7354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:35.929801 7354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:35.929875 7354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:35.962349 7354 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40301
I20260927 22:18:35.962395 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40301 every 8 connection(s)
I20260927 22:18:35.963629 7354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:18:35.969455 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7354
I20260927 22:18:35.969691 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:18:35.970556 24245 external_mini_cluster.cc:1428] Setting key bcc2bc0ea5ab148eb0d4165f06e4a0a9
Sep 27 22:18:35 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547515, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:35.985351 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:35.966130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:56699 (local address 127.23.173.126:41723)
0927 22:18:35.966566 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:35.966571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:35.967445 (+ 874us) server_negotiation.cc:407] Connection header received
0927 22:18:35.969358 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:35.969363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:35.969436 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:35.969531 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:35.971624 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.972742 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.975246 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.975635 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.975729 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:35.979806 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:35.979811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:35.979814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:35.979845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:35.982382 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:35.983112 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:35.983124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:35.983127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:35.983239 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:35.983671 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:35.983676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:35.983679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:35.983932 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:35.984079 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:35.984755 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 22:18:35.984970 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":243,"threads_started":1}
I20260927 22:18:35.986311 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:35.966613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:56699)
0927 22:18:35.967287 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:35.967325 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:35.969038 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:35.969841 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:35.969853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:35.970654 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:35.971415 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:35.971440 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.972934 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.972939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:35.975111 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:35.975128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:35.975860 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:35.975869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:35.976184 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:35.976912 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:35.976948 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:35.979590 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:35.982529 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:35.982537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:35.982557 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:35.982980 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:35.983369 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:35.983374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:35.983377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:35.983556 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:35.984072 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:35.984080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:35.984396 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 22:18:35.984646 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 22:18:35.985158 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260927 22:18:35.987859 7482 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:35.988169 7482 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:35.988868 7482 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:35.991173 7293 ts_manager.cc:194] Registered new tserver with Master: bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:18:35.993489 7293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:56699
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:18:36.025439 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:36.355387 7488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:36.355664 7488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:36.355877 7488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:36.363075 7488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:36.363178 7488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:36.363364 7488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:18:36.372912 7488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:36.374239 7488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:36.375525 7488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:36.388769 7493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:36.388804 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:36.388937 7494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:36.389535 7488 server_base.cc:1034] running on GCE node
I20260927 22:18:36.390196 7488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:36.391125 7488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:36.392354 7488 hybrid_clock.cc:648] HybridClock initialized: now 1790547516392313 us; error 35 us; skew 500 ppm
I20260927 22:18:36.392642 7488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:36 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547516, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:36.395797 7488 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:18:36.397292 7488 webserver.cc:460] Webserver started at http://127.23.173.66:44039/ using document root <none> and password file <none>
I20260927 22:18:36.398087 7488 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:36.398243 7488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:36.398563 7488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:36.401451 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "df78ad053fce4d7297a7360a5c90c03b"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "bb2bded494a6fe62c6b6116c7df4c5ad"
server_key_iv: "095bc52d3f43ddd89e46afe6e66b3e9a"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.402261 7488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "df78ad053fce4d7297a7360a5c90c03b"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "bb2bded494a6fe62c6b6116c7df4c5ad"
server_key_iv: "095bc52d3f43ddd89e46afe6e66b3e9a"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.407423 7488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:18:36.411041 7503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.412511 7488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:18:36.412644 7488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "df78ad053fce4d7297a7360a5c90c03b"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "bb2bded494a6fe62c6b6116c7df4c5ad"
server_key_iv: "095bc52d3f43ddd89e46afe6e66b3e9a"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.412801 7488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:36.429085 7488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:36.433010 7488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:36.433287 7488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:36.434342 7488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:36.435699 7488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:36.435771 7488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.435950 7488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:36.436003 7488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.469597 7488 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:38445
I20260927 22:18:36.469637 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:38445 every 8 connection(s)
I20260927 22:18:36.470995 7488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:18:36.478133 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7488
I20260927 22:18:36.478327 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:18:36.478965 24245 external_mini_cluster.cc:1428] Setting key 9101f4febe8cd448ec9c3b4657deef87
Sep 27 22:18:36 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547516, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:36.492465 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:36.473591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:60349 (local address 127.23.173.126:41723)
0927 22:18:36.473765 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:36.473769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:36.474961 (+ 1192us) server_negotiation.cc:407] Connection header received
0927 22:18:36.476752 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:36.476757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:36.476853 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:36.476968 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:36.478863 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.479629 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.481556 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.482044 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.482179 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:36.486373 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:36.486378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:36.486381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:36.486414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:36.489304 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:36.490078 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:36.490085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:36.490096 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:36.490226 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:36.490679 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:36.490690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:36.490693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:36.490960 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:36.491134 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:36.491847 (+ 713us) server_negotiation.cc:299] Negotiation successful
0927 22:18:36.492104 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:18:36.493448 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:36.474098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:60349)
0927 22:18:36.474789 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:36.474837 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:36.476476 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:36.477215 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:36.477223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:36.477807 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:36.478673 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:36.478703 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.479828 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.479836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:36.481402 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:36.481417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.482297 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.482308 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:36.482652 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:36.483383 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:36.483421 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:36.486134 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:36.489455 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:36.489464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:36.489482 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:36.489924 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:36.490342 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:36.490347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:36.490351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:36.490547 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:36.491101 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:36.491109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:36.491466 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 22:18:36.491754 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:18:36.492327 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":267,"threads_started":1}
I20260927 22:18:36.495046 7616 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:36.495354 7616 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:36.496074 7616 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:36.497517 7293 ts_manager.cc:194] Registered new tserver with Master: df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445)
I20260927 22:18:36.498400 7293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:60349
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:18:36.534798 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:36.855569 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:36.855836 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:36.855937 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:36.862766 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:36.862888 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:36.863008 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:18:36.872303 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:36.873508 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:36.874787 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:36.887713 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:36.887665 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:36.887667 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:36.888643 7621 server_base.cc:1034] running on GCE node
I20260927 22:18:36.889330 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:36.890332 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:36.891557 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790547516891503 us; error 48 us; skew 500 ppm
I20260927 22:18:36.891877 7621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:36 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547516, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:36.894995 7621 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:18:36.896454 7621 webserver.cc:460] Webserver started at http://127.23.173.67:34135/ using document root <none> and password file <none>
I20260927 22:18:36.897131 7621 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:36.897271 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:36.897572 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:18:36.900494 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "fafe4dd559b64f9098b2be14bfdbf381"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "36d259c1fbd15f0ca77c988018927052"
server_key_iv: "2f8db75b19aa4200126b7c5b8f2b435b"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.901340 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "fafe4dd559b64f9098b2be14bfdbf381"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "36d259c1fbd15f0ca77c988018927052"
server_key_iv: "2f8db75b19aa4200126b7c5b8f2b435b"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.906420 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:18:36.909725 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.911331 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:36.911499 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fafe4dd559b64f9098b2be14bfdbf381"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "36d259c1fbd15f0ca77c988018927052"
server_key_iv: "2f8db75b19aa4200126b7c5b8f2b435b"
server_key_version: "encryptionkey@0"
I20260927 22:18:36.911697 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:36.938653 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:36.942584 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:36.942894 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:36.943856 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:36.945191 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:36.945291 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.945436 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:36.945508 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:36.978643 7621 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:46091
I20260927 22:18:36.978685 7748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:46091 every 8 connection(s)
I20260927 22:18:36.979872 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:18:36.987684 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7621
I20260927 22:18:36.987876 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:18:36.988324 24245 external_mini_cluster.cc:1428] Setting key 1cf873ebd1fb75268d56b2aa32b85a78
Sep 27 22:18:36 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547516, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:36.996454 7482 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:18:36.999855 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:36.982450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:49707 (local address 127.23.173.126:41723)
0927 22:18:36.982647 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:36.982652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:36.983931 (+ 1279us) server_negotiation.cc:407] Connection header received
0927 22:18:36.985796 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:36.985801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:36.985901 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:36.986099 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:36.987551 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.988465 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.990299 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.990921 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.991059 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:36.994349 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:36.994353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:36.994357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:36.994387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:36.996792 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:36.997636 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:36.997640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:36.997643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:36.997810 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:36.998428 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:36.998431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:36.998434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:36.998688 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:36.998800 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:36.999431 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 22:18:36.999589 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:18:37.000821 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:36.982886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:49707)
0927 22:18:36.983771 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:36.983810 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:36.985524 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:36.986314 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:36.986323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:36.986800 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:36.987362 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:36.987387 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.988623 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.988629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:36.990133 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:36.990158 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:36.991122 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:36.991127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:36.991308 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:36.991977 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:36.991998 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:36.994168 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:36.996927 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:36.996935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:36.996950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:36.997526 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:36.998136 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:36.998139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:36.998141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:36.998328 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:36.998804 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:36.998809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:36.999085 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 22:18:36.999314 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 22:18:36.999681 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":400,"threads_started":1}
I20260927 22:18:37.002203 7749 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:37.002571 7749 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:37.003408 7749 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:37.004868 7293 ts_manager.cc:194] Registered new tserver with Master: fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67:46091)
I20260927 22:18:37.005658 7293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:49707
I20260927 22:18:37.016098 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:18:37.032845 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.020655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.23.173.126:41723)
0927 22:18:37.020814 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.020818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.021219 (+ 401us) server_negotiation.cc:407] Connection header received
0927 22:18:37.021356 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.021360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.021419 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.021504 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:37.022953 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.023776 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.025099 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.025512 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.025613 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.028006 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:37.028010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:37.028013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:37.028050 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:37.030701 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.031294 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.031297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.031299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.031423 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.031736 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.031739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.031741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.032046 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:37.032237 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.032483 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.032628 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:18:37.036520 7293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:18:37.043315 7293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:18:37.065181 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.056697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.23.173.65:40301)
0927 22:18:37.057289 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.057297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.058882 (+ 1585us) server_negotiation.cc:407] Connection header received
0927 22:18:37.058982 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.058989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.059177 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.059368 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:37.059900 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.060898 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.063161 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.064083 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.064228 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.064477 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.064710 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.064866 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":372,"threads_started":1}
I20260927 22:18:37.065213 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.056938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40301 (local address 127.0.0.1:35942)
0927 22:18:37.058623 (+ 1685us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:37.058642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:37.058766 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:37.059364 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:37.059369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:37.059394 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:37.059731 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.059742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.061045 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.061051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.063001 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.063014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.064275 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.064281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.064467 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.064524 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:18:37.064667 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:18:37.064848 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":1374,"threads_started":1}
I20260927 22:18:37.065986 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.054849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:46091 (local address 127.0.0.1:37208)
0927 22:18:37.055998 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:37.056035 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:37.056207 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:37.057494 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:37.057498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:37.057518 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:37.058123 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.058137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.059926 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.059931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.061626 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.061642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.064887 (+ 3245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.064895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.065086 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.065131 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:18:37.065199 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 22:18:37.065580 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":745,"threads_started":1}
I20260927 22:18:37.066875 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.055317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.23.173.67:46091)
0927 22:18:37.056870 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.056878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.056910 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:18:37.057011 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.057019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.057268 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.057501 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:37.058317 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.059798 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.061766 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.064649 (+ 2883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.066083 (+ 1434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.066292 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.066361 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.066511 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":1210,"threads_started":1}
I20260927 22:18:37.067366 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.054301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38445 (local address 127.0.0.1:39758)
0927 22:18:37.055989 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:37.056033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:37.056270 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:37.057274 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:37.057282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:37.057320 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:37.057974 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.057993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.061071 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.061081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.063288 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.063300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.066550 (+ 3250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.066556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.066737 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.066772 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:18:37.066876 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:18:37.067001 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":389,"thread_start_us":1443,"threads_started":1}
I20260927 22:18:37.067425 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.054484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.23.173.66:38445)
0927 22:18:37.055510 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.055523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.056171 (+ 648us) server_negotiation.cc:407] Connection header received
0927 22:18:37.056510 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.056521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.056786 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.057011 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:37.058203 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.059762 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.063441 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.066087 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.066221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.066405 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.066945 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.067147 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":687,"threads_started":1}
I20260927 22:18:37.069765 7417 tablet_service.cc:1467] Processing CreateTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 (DEFAULT_TABLE table=test-table [id=f7ced160bfd946599f91fe679a32ddd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:37.070483 7684 tablet_service.cc:1467] Processing CreateTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 (DEFAULT_TABLE table=test-table [id=f7ced160bfd946599f91fe679a32ddd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:37.070640 7551 tablet_service.cc:1467] Processing CreateTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 (DEFAULT_TABLE table=test-table [id=f7ced160bfd946599f91fe679a32ddd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:18:37.071323 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add92bf9f41e4eb39e8f13d7e7d10d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:37.071744 7551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add92bf9f41e4eb39e8f13d7e7d10d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:37.071864 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add92bf9f41e4eb39e8f13d7e7d10d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:18:37.082695 7767 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Bootstrap starting.
I20260927 22:18:37.085793 7768 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Bootstrap starting.
I20260927 22:18:37.086047 7767 tablet_bootstrap.cc:654] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:37.088074 7767 log.cc:824] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:37.089313 7768 tablet_bootstrap.cc:654] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:37.090162 7769 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Bootstrap starting.
I20260927 22:18:37.090665 7768 log.cc:824] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:37.091503 7767 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: No bootstrap required, opened a new log
I20260927 22:18:37.091770 7767 ts_tablet_manager.cc:1397] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:18:37.093062 7769 tablet_bootstrap.cc:654] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Neither blocks nor log segments found. Creating new log.
I20260927 22:18:37.093693 7768 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: No bootstrap required, opened a new log
I20260927 22:18:37.093948 7768 ts_tablet_manager.cc:1397] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:18:37.094190 7769 log.cc:824] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:37.097241 7769 tablet_bootstrap.cc:492] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: No bootstrap required, opened a new log
I20260927 22:18:37.097492 7769 ts_tablet_manager.cc:1397] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:18:37.102315 7767 raft_consensus.cc:348] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.102744 7767 raft_consensus.cc:374] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:37.102873 7767 raft_consensus.cc:729] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78ad053fce4d7297a7360a5c90c03b, State: Initialized, Role: FOLLOWER
I20260927 22:18:37.103497 7767 consensus_queue.cc:260] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.104525 7768 raft_consensus.cc:348] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.105026 7768 raft_consensus.cc:374] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:37.104809 7769 raft_consensus.cc:348] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.105144 7768 raft_consensus.cc:729] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafe4dd559b64f9098b2be14bfdbf381, State: Initialized, Role: FOLLOWER
I20260927 22:18:37.105197 7769 raft_consensus.cc:374] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:18:37.105299 7769 raft_consensus.cc:729] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8a7e57d8344f7b937f818fccdab501, State: Initialized, Role: FOLLOWER
I20260927 22:18:37.105837 7768 consensus_queue.cc:260] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.105960 7769 consensus_queue.cc:260] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.110764 7616 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:18:37.111486 7769 ts_tablet_manager.cc:1428] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260927 22:18:37.111922 7767 ts_tablet_manager.cc:1428] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260927 22:18:37.112825 7749 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:18:37.113425 7768 ts_tablet_manager.cc:1428] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260927 22:18:37.215956 7483 tablet.cc:2369] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:37.223093 7617 tablet.cc:2369] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:18:37.231455 7750 tablet.cc:2369] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:18:37.334832 7774 raft_consensus.cc:482] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:18:37.335198 7774 raft_consensus.cc:504] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.336850 7774 leader_election.cc:288] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445), bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:18:37.343499 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.337310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:48987 (local address 127.23.173.66:38445)
0927 22:18:37.337546 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.337555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.337580 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:18:37.337800 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.337805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.337891 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.338028 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:37.338541 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.339604 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.341478 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.342643 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.342790 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.342842 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.343221 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.343311 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:18:37.343626 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.337205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38445 (local address 127.23.173.67:48987)
0927 22:18:37.337429 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:37.337450 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:37.337590 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:37.338135 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:37.338141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:37.338171 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:37.338341 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.338349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.339740 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.339746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.341303 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.341315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.342815 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.342821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.343029 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.343076 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:18:37.343179 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:18:37.343316 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":25}
I20260927 22:18:37.344399 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.337794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:54863 (local address 127.23.173.65:40301)
0927 22:18:37.338027 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.338047 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.338431 (+ 384us) server_negotiation.cc:407] Connection header received
0927 22:18:37.338594 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.338598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.338655 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.338723 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:18:37.339339 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.340319 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.342375 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.343458 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.343632 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.343717 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.344017 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.344162 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:18:37.344427 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.337626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40301 (local address 127.23.173.67:54863)
0927 22:18:37.338321 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:37.338335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:37.338459 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:37.338775 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:37.338780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:37.338807 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:18:37.339195 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.339206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.340469 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.340473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.342229 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:37.342244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.343705 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.343712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:37.343865 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.343888 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:18:37.343962 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 22:18:37.344103 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":18,"thread_start_us":386,"threads_started":1}
I20260927 22:18:37.344889 7571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add92bf9f41e4eb39e8f13d7e7d10d01" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78ad053fce4d7297a7360a5c90c03b" is_pre_election: true
I20260927 22:18:37.345461 7571 raft_consensus.cc:2454] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 0.
I20260927 22:18:37.345613 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add92bf9f41e4eb39e8f13d7e7d10d01" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8a7e57d8344f7b937f818fccdab501" is_pre_election: true
I20260927 22:18:37.346179 7638 leader_election.cc:302] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df78ad053fce4d7297a7360a5c90c03b, fafe4dd559b64f9098b2be14bfdbf381; no voters:
I20260927 22:18:37.346194 7437 raft_consensus.cc:2454] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 0.
I20260927 22:18:37.346550 7774 raft_consensus.cc:2781] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:18:37.346704 7774 raft_consensus.cc:482] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:18:37.346817 7774 raft_consensus.cc:3037] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:37.348984 7774 raft_consensus.cc:504] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.349676 7774 leader_election.cc:288] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 election: Requested vote from peers df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445), bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:18:37.350050 7571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add92bf9f41e4eb39e8f13d7e7d10d01" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78ad053fce4d7297a7360a5c90c03b"
I20260927 22:18:37.350160 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add92bf9f41e4eb39e8f13d7e7d10d01" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8a7e57d8344f7b937f818fccdab501"
I20260927 22:18:37.350333 7571 raft_consensus.cc:3037] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:37.350409 7437 raft_consensus.cc:3037] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:18:37.353569 7571 raft_consensus.cc:2454] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 1.
I20260927 22:18:37.353569 7437 raft_consensus.cc:2454] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 1.
I20260927 22:18:37.354107 7638 leader_election.cc:302] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8a7e57d8344f7b937f818fccdab501, fafe4dd559b64f9098b2be14bfdbf381; no voters:
I20260927 22:18:37.354427 7774 raft_consensus.cc:2781] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:18:37.354867 7774 raft_consensus.cc:686] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 1 LEADER]: Becoming Leader. State: Replica: fafe4dd559b64f9098b2be14bfdbf381, State: Running, Role: LEADER
I20260927 22:18:37.355326 7774 consensus_queue.cc:237] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } }
I20260927 22:18:37.360126 7291 catalog_manager.cc:5495] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 reported cstate change: term changed from 0 to 1, leader changed from <none> to fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "fafe4dd559b64f9098b2be14bfdbf381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } health_report { overall_health: HEALTHY } } }
I20260927 22:18:37.437496 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.432007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.23.173.67:46091)
0927 22:18:37.432212 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.432218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.432240 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:18:37.432449 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.432452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.432531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.432693 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:37.433107 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.433830 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.435281 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.435706 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.435848 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.436390 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:37.436917 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:18:37.437085 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.437147 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.437229 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:18:37.445693 7571 raft_consensus.cc:1261] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Refusing update from remote peer fafe4dd559b64f9098b2be14bfdbf381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:37.445693 7437 raft_consensus.cc:1261] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Refusing update from remote peer fafe4dd559b64f9098b2be14bfdbf381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:18:37.446766 7777 consensus_queue.cc:1035] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:37.447217 7774 consensus_queue.cc:1035] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:18:37.471370 7291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:37.471668 7291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:18:37.475116 7291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Sending DeleteTablet for 3 replicas of tablet add92bf9f41e4eb39e8f13d7e7d10d01
I20260927 22:18:37.476229 7417 tablet_service.cc:1514] Processing DeleteTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:37 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35942
I20260927 22:18:37.476234 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:37 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37208
I20260927 22:18:37.477322 7551 tablet_service.cc:1514] Processing DeleteTablet for tablet add92bf9f41e4eb39e8f13d7e7d10d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:18:37 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:39758
I20260927 22:18:37.479846 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7354
I20260927 22:18:37.487934 7791 tablet_replica.cc:330] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: stopping tablet replica
I20260927 22:18:37.489120 7791 raft_consensus.cc:2229] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:37.489673 7791 raft_consensus.cc:2258] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:37.491866 7790 tablet_replica.cc:330] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: stopping tablet replica
I20260927 22:18:37.492307 7790 raft_consensus.cc:2229] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:18:37.492997 7790 raft_consensus.cc:2258] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:18:37.495175 7790 ts_tablet_manager.cc:1905] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:37.495435 7791 ts_tablet_manager.cc:1905] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:37.503012 7792 tablet_replica.cc:330] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: stopping tablet replica
I20260927 22:18:37.503432 7792 raft_consensus.cc:2229] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:18:37.504002 7792 raft_consensus.cc:2258] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:18:37 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:18:37.506318 7792 ts_tablet_manager.cc:1905] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:18:37.506613 7791 ts_tablet_manager.cc:1918] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:37.506825 7791 log.cc:1196] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/add92bf9f41e4eb39e8f13d7e7d10d01
I20260927 22:18:37.507426 7791 ts_tablet_manager.cc:1939] T add92bf9f41e4eb39e8f13d7e7d10d01 P bd8a7e57d8344f7b937f818fccdab501: Deleting consensus metadata
I20260927 22:18:37.509476 7278 catalog_manager.cc:4841] TS bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301): tablet add92bf9f41e4eb39e8f13d7e7d10d01 (table test-table [id=f7ced160bfd946599f91fe679a32ddd1]) successfully deleted
I20260927 22:18:37.511070 7790 ts_tablet_manager.cc:1918] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:37.511289 7790 log.cc:1196] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/add92bf9f41e4eb39e8f13d7e7d10d01
I20260927 22:18:37.511907 7790 ts_tablet_manager.cc:1939] T add92bf9f41e4eb39e8f13d7e7d10d01 P fafe4dd559b64f9098b2be14bfdbf381: Deleting consensus metadata
I20260927 22:18:37.513715 7277 catalog_manager.cc:4841] TS fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67:46091): tablet add92bf9f41e4eb39e8f13d7e7d10d01 (table test-table [id=f7ced160bfd946599f91fe679a32ddd1]) successfully deleted
I20260927 22:18:37.514360 7792 ts_tablet_manager.cc:1918] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:18:37.514580 7792 log.cc:1196] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/add92bf9f41e4eb39e8f13d7e7d10d01
I20260927 22:18:37.515120 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.481253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.23.173.65:40301)
0927 22:18:37.481473 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.481483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.494508 (+ 13025us) server_negotiation.cc:407] Connection header received
0927 22:18:37.494806 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.494812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.494895 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.494978 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:37.497013 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.498173 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.500333 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.500991 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.501089 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.507151 (+ 6062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:37.507184 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:37.507204 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:37.507246 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:37.511265 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.512349 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.512357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.512360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.512531 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.513061 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.513066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.513069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.513534 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:37.513798 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.514461 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.514737 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:18:37.515142 7792 ts_tablet_manager.cc:1939] T add92bf9f41e4eb39e8f13d7e7d10d01 P df78ad053fce4d7297a7360a5c90c03b: Deleting consensus metadata
I20260927 22:18:37.515889 7477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:37.517338 7278 catalog_manager.cc:4841] TS df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445): tablet add92bf9f41e4eb39e8f13d7e7d10d01 (table test-table [id=f7ced160bfd946599f91fe679a32ddd1]) successfully deleted
I20260927 22:18:37.606714 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7354
I20260927 22:18:37.627382 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7488
Sep 27 22:18:37 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:18:37.642653 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.628227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.23.173.66:38445)
0927 22:18:37.628401 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.628407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.628473 (+ 66us) server_negotiation.cc:407] Connection header received
0927 22:18:37.628676 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.628682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.628752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.628867 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:37.630208 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.630915 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.632306 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.632755 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.632892 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.637052 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:37.637068 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:37.637083 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:37.637117 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:37.639896 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.640544 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.640551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.640553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.640685 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.641061 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.641067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.641073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.641414 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:37.641723 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.642138 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.642345 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:18:37.643311 7611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:37.733878 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7488
I20260927 22:18:37.758069 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7621
Sep 27 22:18:37 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:18:37.776502 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:37.759029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.23.173.67:46091)
0927 22:18:37.759236 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:37.759243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:37.759293 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:18:37.759362 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:37.759368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:37.759478 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:37.759609 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:37.760994 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.762205 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.763692 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:37.764405 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:37.764556 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:37.769152 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:37.769169 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:37.769189 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:37.769236 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:37.773390 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.774227 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.774237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.774240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.774441 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:37.774814 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:37.774819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:37.774823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:37.775321 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:37.775660 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:37.775841 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 22:18:37.776073 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:18:37.777427 7744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:37.900415 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7621
I20260927 22:18:37.920333 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7261
I20260927 22:18:37.921485 7323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:18:38.032866 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7261
I20260927 22:18:58.056523 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--webserver_interface=127.23.173.126
--webserver_port=45617
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:18:58.384972 7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:58.385257 7798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:58.385344 7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:58.392562 7798 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:18:58.392652 7798 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:18:58.392689 7798 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:18:58.392733 7798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:18:58.392777 7798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:58.392844 7798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:58.392889 7798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:18:58.392935 7798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:18:58.402688 7798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=45617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:58.403875 7798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:58.405148 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:58.427947 7807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:58.428053 7805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:58.428053 7804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:58.428790 7798 server_base.cc:1034] running on GCE node
I20260927 22:18:58.429750 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:58.431444 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:58.432708 7798 hybrid_clock.cc:648] HybridClock initialized: now 1790547538432643 us; error 48 us; skew 500 ppm
I20260927 22:18:58.433110 7798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:58 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547538, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:58.440949 7798 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:18:58.442752 7798 webserver.cc:460] Webserver started at http://127.23.173.126:45617/ using document root <none> and password file <none>
I20260927 22:18:58.443531 7798 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:58.443630 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:58.449536 7798 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:18:58.452565 7814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:58.454025 7798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:18:58.454226 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c557d0b9625744e48c9ae3ad4fddf023"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "87a8d92ce4cd2d1d2b32f0c82d39e874"
server_key_iv: "b0717db577e7c8e03e232db468b88a2f"
server_key_version: "encryptionkey@0"
I20260927 22:18:58.455289 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:58.482291 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:58.486413 7798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:58.486714 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:58.505687 7798 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:41723
I20260927 22:18:58.505728 7865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:41723 every 8 connection(s)
I20260927 22:18:58.507226 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:18:58.512890 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7798
I20260927 22:18:58.513935 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:40301
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=45309
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:18:58.514700 7866 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:18:58.519878 7866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap starting.
I20260927 22:18:58.540477 7866 log.cc:824] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Log is configured to *not* fsync() on all Append() calls
I20260927 22:18:58.552434 7866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:18:58.552976 7866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap complete.
I20260927 22:18:58.562419 7866 raft_consensus.cc:348] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:58.563266 7866 raft_consensus.cc:729] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Initialized, Role: FOLLOWER
I20260927 22:18:58.563849 7866 consensus_queue.cc:260] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:58.564103 7866 raft_consensus.cc:388] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:18:58.564241 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:18:58.564422 7866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:18:58.568183 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:58.568710 7866 leader_election.cc:302] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c557d0b9625744e48c9ae3ad4fddf023; no voters:
I20260927 22:18:58.569367 7866 leader_election.cc:288] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:18:58.569499 7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:18:58.570739 7870 raft_consensus.cc:686] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 LEADER]: Becoming Leader. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Running, Role: LEADER
I20260927 22:18:58.571380 7870 consensus_queue.cc:237] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:18:58.571709 7866 sys_catalog.cc:564] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:18:58.574172 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c557d0b9625744e48c9ae3ad4fddf023. Latest consensus state: current_term: 2 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:18:58.574180 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:18:58.574568 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:58.574651 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:18:58.583375 7876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:18:58.593024 7876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:18:58.594911 7876 catalog_manager.cc:1259] Loaded cluster ID: 595a15ac28af47349ccd5a015f35eca3
I20260927 22:18:58.595084 7876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:18:58.599956 7876 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:18:58.601927 7876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Loaded TSK: 0
I20260927 22:18:58.641105 7876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Generated new TSK 1
I20260927 22:18:58.641508 7876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:18:58.876340 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:58.876608 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:58.876754 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:58.884194 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:58.884305 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:58.884487 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:18:58.894341 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:40301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=45309
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:58.895596 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:58.896878 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:58.910306 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:58.910346 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:58.914805 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:58.915699 7868 server_base.cc:1034] running on GCE node
I20260927 22:18:58.916254 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:58.917194 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:58.918466 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790547538918415 us; error 37 us; skew 500 ppm
I20260927 22:18:58.918864 7868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:58 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547538, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:58.923115 7868 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:18:58.924918 7868 webserver.cc:460] Webserver started at http://127.23.173.65:45309/ using document root <none> and password file <none>
I20260927 22:18:58.925880 7868 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:58.926079 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:58.933251 7868 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 22:18:58.936664 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:58.938459 7868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:58.938622 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "bd8a7e57d8344f7b937f818fccdab501"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "96e896248f813ea49afe3c752cce8a83"
server_key_iv: "5f0e245e51a700748a62b2d47bebf3a1"
server_key_version: "encryptionkey@0"
I20260927 22:18:58.939575 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:58.963110 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:58.967490 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:58.967789 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:58.968789 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:58.970314 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:58.970404 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:58.970553 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:58.970659 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.005060 7868 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40301
I20260927 22:18:59.005147 8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40301 every 8 connection(s)
I20260927 22:18:59.006486 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:18:59.011312 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 7868
I20260927 22:18:59.012065 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:38445
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=44039
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:18:59 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547538, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:59.035513 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.009353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:33765 (local address 127.23.173.126:41723)
0927 22:18:59.010023 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.010043 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.010564 (+ 521us) server_negotiation.cc:407] Connection header received
0927 22:18:59.013653 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.013666 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.014074 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.014377 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:59.016356 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.020218 (+ 3862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.022788 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.023430 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.023559 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.027979 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:59.027986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:59.028001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:59.028039 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:59.031004 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.031993 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.031999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.032004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.032158 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.032706 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.032710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.032712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.033127 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:59.033282 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:59.034337 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0927 22:18:59.034627 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
I20260927 22:18:59.036414 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.009643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:33765)
0927 22:18:59.010391 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:59.010427 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:59.013090 (+ 2663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:59.014600 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:59.014614 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:59.015365 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:59.016131 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.016158 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.020395 (+ 4237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.020402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.022640 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.022658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.023702 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.023713 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.024023 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.024715 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:59.024759 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:59.027702 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:59.031147 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.031157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.031172 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.031847 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.032297 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.032305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.032310 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.032582 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.033303 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:59.033312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:59.033791 (+ 479us) client_negotiation.cc:769] Sending connection context
0927 22:18:59.034148 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:18:59.034763 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260927 22:18:59.039441 8016 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:59.039757 8016 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:59.040537 8016 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:59.042907 7831 ts_manager.cc:194] Registered new tserver with Master: bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:18:59.044428 7831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:33765
W20260927 22:18:59.341763 8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:59.342087 8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:59.342262 8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:59.349483 8020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:59.349577 8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:59.349741 8020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:18:59.359318 8020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=44039
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:59.360612 8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:59.361891 8020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:59.374963 8026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:59.375360 8028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:59.374990 8025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:59.375748 8020 server_base.cc:1034] running on GCE node
I20260927 22:18:59.376333 8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:59.377203 8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:59.378407 8020 hybrid_clock.cc:648] HybridClock initialized: now 1790547539378371 us; error 32 us; skew 500 ppm
I20260927 22:18:59.378736 8020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:59 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547539, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:59.381724 8020 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:18:59.383217 8020 webserver.cc:460] Webserver started at http://127.23.173.66:44039/ using document root <none> and password file <none>
I20260927 22:18:59.383867 8020 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:59.384018 8020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:59.389025 8020 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 22:18:59.391675 8035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.392915 8020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:18:59.393100 8020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "df78ad053fce4d7297a7360a5c90c03b"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "bb2bded494a6fe62c6b6116c7df4c5ad"
server_key_iv: "095bc52d3f43ddd89e46afe6e66b3e9a"
server_key_version: "encryptionkey@0"
I20260927 22:18:59.394037 8020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:59.414662 8020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:59.418411 8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:59.418673 8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:59.419643 8020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:59.420902 8020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:59.421031 8020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.421177 8020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:59.421267 8020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.454777 8020 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:38445
I20260927 22:18:59.454862 8147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:38445 every 8 connection(s)
I20260927 22:18:59.456213 8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:18:59.458549 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8020
I20260927 22:18:59.459262 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:46091
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=34135
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:18:59 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547539, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:59.478097 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.458962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38073 (local address 127.23.173.126:41723)
0927 22:18:59.459176 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.459183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.460619 (+ 1436us) server_negotiation.cc:407] Connection header received
0927 22:18:59.462589 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.462598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.462703 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.462833 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:59.464311 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.465444 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.467677 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.468085 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.468182 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.472355 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:59.472360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:59.472363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:59.472398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:59.474956 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.475703 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.475709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.475711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.475831 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.476279 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.476283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.476285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.476557 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:59.476727 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:59.477486 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 22:18:59.477727 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:18:59.479053 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.459577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:38073)
0927 22:18:59.460446 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:59.460491 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:59.462311 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:59.463052 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:59.463069 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:59.463578 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:59.464132 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.464149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.465598 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.465601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.467525 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.467543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.468317 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.468326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.468648 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.469367 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:59.469405 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:59.472076 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:59.475098 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.475107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.475124 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.475551 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.475951 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.475956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.475960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.476143 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.476687 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:59.476695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:59.477061 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 22:18:59.477389 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 22:18:59.477907 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":389,"threads_started":1}
I20260927 22:18:59.480911 8148 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:59.481431 8148 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:59.482476 8148 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:59.484375 7831 ts_manager.cc:194] Registered new tserver with Master: df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445)
I20260927 22:18:59.485699 7831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:38073
W20260927 22:18:59.801403 8151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:18:59.801669 8151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:18:59.801762 8151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:18:59.808928 8151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:18:59.809051 8151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:18:59.809190 8151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:18:59.818724 8151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:46091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=34135
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:18:59.819957 8151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:18:59.821211 8151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:18:59.834116 8156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:59.834091 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:18:59.834280 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:18:59.834838 8151 server_base.cc:1034] running on GCE node
I20260927 22:18:59.835489 8151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:18:59.836398 8151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:18:59.837620 8151 hybrid_clock.cc:648] HybridClock initialized: now 1790547539837577 us; error 58 us; skew 500 ppm
I20260927 22:18:59.837939 8151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:18:59 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547539, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:18:59.841027 8151 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:18:59.842468 8151 webserver.cc:460] Webserver started at http://127.23.173.67:34135/ using document root <none> and password file <none>
I20260927 22:18:59.843226 8151 fs_manager.cc:359] Metadata directory not provided
I20260927 22:18:59.843366 8151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:18:59.848628 8151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 22:18:59.851249 8166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.852550 8151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:18:59.852730 8151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fafe4dd559b64f9098b2be14bfdbf381"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "36d259c1fbd15f0ca77c988018927052"
server_key_iv: "2f8db75b19aa4200126b7c5b8f2b435b"
server_key_version: "encryptionkey@0"
I20260927 22:18:59.853648 8151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:18:59.888120 8151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:18:59.892437 8151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:18:59.892705 8151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:18:59.893642 8151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:18:59.894965 8151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:18:59.895084 8151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.895223 8151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:18:59.895305 8151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:18:59.928511 8151 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:46091
I20260927 22:18:59.928565 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:46091 every 8 connection(s)
I20260927 22:18:59.929949 8151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:18:59.934775 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8151
Sep 27 22:18:59 dist-test-slave-3wc3 krb5kdc[7248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547539, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:18:59.950582 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.935846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.23.173.126:41723)
0927 22:18:59.936572 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.936580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.936600 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:18:59.936659 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.936662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.936713 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.936824 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:59.938453 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.939312 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.940574 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.941198 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.941289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.944721 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:59.944726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:59.944729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:59.944758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:59.947916 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.948594 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.948601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.948605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.948740 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.949112 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.949118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.949121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.949383 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:59.949672 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:59.949937 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:18:59.950194 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":621,"thread_start_us":497,"threads_started":1}
I20260927 22:18:59.951274 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.932585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:48283 (local address 127.23.173.126:41723)
0927 22:18:59.932812 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.932819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.933990 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 22:18:59.936483 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.936488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.936552 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.936607 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:59.938272 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.939101 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.940701 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.941288 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.941416 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.945181 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:59.945186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:59.945189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:59.945233 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:59.948104 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.948816 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.948823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.948831 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.948946 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.949366 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.949371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.949374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.949622 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:59.949776 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:59.950683 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 22:18:59.950900 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":83}
I20260927 22:18:59.952478 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.933092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:48283)
0927 22:18:59.933787 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:18:59.933828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:18:59.936086 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:18:59.936998 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:18:59.937006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:18:59.937492 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:18:59.938099 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.938118 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.939274 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.939280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.940557 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:18:59.940570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.941520 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.941526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:18:59.941715 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.942504 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:18:59.942531 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:18:59.944978 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:18:59.948245 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.948254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.948269 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.948688 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.949054 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:18:59.949060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:18:59.949063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:18:59.949244 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:18:59.949756 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:18:59.949764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:18:59.950278 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 22:18:59.950583 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 22:18:59.951232 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":276,"threads_started":1}
I20260927 22:18:59.953797 8279 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41723
I20260927 22:18:59.954104 8279 heartbeater.cc:464] Registering TS with master...
I20260927 22:18:59.954747 8279 heartbeater.cc:511] Master 127.23.173.126:41723 requested a full tablet report, sending...
I20260927 22:18:59.955988 7831 ts_manager.cc:194] Registered new tserver with Master: fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67:46091)
I20260927 22:18:59.956768 7831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:48283
I20260927 22:18:59.965062 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:18:59.971457 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.965879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.23.173.126:41723)
0927 22:18:59.966089 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.966093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.966633 (+ 540us) server_negotiation.cc:407] Connection header received
0927 22:18:59.966835 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.966838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.966961 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.967078 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:18:59.967484 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.968226 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.969763 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.970224 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.970317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.970742 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:18:59.970924 (+ 182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:18:59.971189 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59648: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:18:59.971561 8018 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59648: token expired
W20260927 22:18:59.972139 7753 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.173.126:41723)
I20260927 22:18:59.985030 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:18:59.972629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.23.173.126:41723)
0927 22:18:59.972793 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:18:59.972799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:18:59.972907 (+ 108us) server_negotiation.cc:407] Connection header received
0927 22:18:59.973075 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:18:59.973080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:18:59.973158 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:18:59.973253 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:18:59.974641 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.975334 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.976728 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:18:59.977106 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:18:59.977242 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:18:59.979825 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:18:59.979830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:18:59.979833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:18:59.979864 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:18:59.982362 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.983137 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.983150 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.983154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.983276 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:18:59.983648 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:18:59.983654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:18:59.983659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:18:59.983953 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:18:59.984208 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:18:59.984481 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:18:59.984691 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:18:59.988256 7831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:18:59.995357 7831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:00.023190 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.006987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38445 (local address 127.0.0.1:39630)
0927 22:19:00.013575 (+ 6588us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:00.013609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:00.013788 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:00.015252 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:00.015258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:00.015283 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:00.015635 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.015653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.017901 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.017907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.020241 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.020256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.022089 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.022095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.022278 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.022336 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:19:00.022515 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:19:00.022715 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6367,"mutex_wait_us":98,"thread_start_us":6213,"threads_started":1}
I20260927 22:19:00.023605 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.007736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.23.173.67:46091)
0927 22:19:00.012983 (+ 5247us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.012996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.014517 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 22:19:00.014796 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.014802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.015042 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.015214 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:00.016373 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.017666 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.020591 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.021896 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.022027 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.022256 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.023076 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.023238 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4995,"thread_start_us":4811,"threads_started":1}
I20260927 22:19:00.023880 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.014264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.23.173.65:40301)
0927 22:19:00.014874 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.014883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.016277 (+ 1394us) server_negotiation.cc:407] Connection header received
0927 22:19:00.016568 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.016576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.016797 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.016950 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:00.017715 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.019360 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.021355 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.022516 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.022635 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.022840 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.023438 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.023580 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":302,"threads_started":1}
I20260927 22:19:00.023883 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.007151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.23.173.66:38445)
0927 22:19:00.014652 (+ 7501us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.014663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.014690 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:19:00.014794 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.014801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.015038 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.015223 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:00.015920 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.017734 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.020395 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.021840 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.022018 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.022221 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.023418 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.023580 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7314,"thread_start_us":7559,"threads_started":1}
I20260927 22:19:00.024592 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.014241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40301 (local address 127.0.0.1:52424)
0927 22:19:00.016173 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:00.016190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:00.016383 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:00.017225 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:00.017229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:00.017252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:00.017560 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.017569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.019522 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.019527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.021207 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.021218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.023016 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.023024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.023230 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.023267 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:19:00.024054 (+ 787us) client_negotiation.cc:240] Negotiation successful
0927 22:19:00.024189 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"mutex_wait_us":1164,"thread_start_us":418,"threads_started":1}
I20260927 22:19:00.025027 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.007079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:46091 (local address 127.0.0.1:44568)
0927 22:19:00.014407 (+ 7328us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:00.014427 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:00.014615 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:00.015510 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:00.015517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:00.015552 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:00.016162 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.016176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.017874 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.017879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.020427 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.020441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.022432 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.022439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.022835 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.022876 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:19:00.024282 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0927 22:19:00.024416 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7194,"mutex_wait_us":18,"thread_start_us":7041,"threads_started":1}
I20260927 22:19:00.027629 8083 tablet_service.cc:1467] Processing CreateTablet for tablet 73c07d1954ec4c27aec682d99778bb5a (DEFAULT_TABLE table=test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:00.027933 7951 tablet_service.cc:1467] Processing CreateTablet for tablet 73c07d1954ec4c27aec682d99778bb5a (DEFAULT_TABLE table=test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:00.028925 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c07d1954ec4c27aec682d99778bb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:00.029279 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c07d1954ec4c27aec682d99778bb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:00.029279 8214 tablet_service.cc:1467] Processing CreateTablet for tablet 73c07d1954ec4c27aec682d99778bb5a (DEFAULT_TABLE table=test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:00.030568 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c07d1954ec4c27aec682d99778bb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:00.041703 8295 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Bootstrap starting.
I20260927 22:19:00.044060 8296 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Bootstrap starting.
I20260927 22:19:00.046200 8295 tablet_bootstrap.cc:654] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:00.047535 8296 tablet_bootstrap.cc:654] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:00.047606 8295 log.cc:824] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:00.047871 8016 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:19:00.048995 8296 log.cc:824] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:00.050944 8297 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Bootstrap starting.
I20260927 22:19:00.058288 8296 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: No bootstrap required, opened a new log
I20260927 22:19:00.058704 8296 ts_tablet_manager.cc:1397] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 22:19:00.055977 8297 tablet_bootstrap.cc:654] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:00.067381 8297 log.cc:824] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:00.068907 8295 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: No bootstrap required, opened a new log
I20260927 22:19:00.069208 8295 ts_tablet_manager.cc:1397] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.013s
I20260927 22:19:00.069876 8297 tablet_bootstrap.cc:492] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: No bootstrap required, opened a new log
I20260927 22:19:00.070144 8297 ts_tablet_manager.cc:1397] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260927 22:19:00.070964 8296 raft_consensus.cc:348] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.071386 8296 raft_consensus.cc:374] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:00.071528 8296 raft_consensus.cc:729] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8a7e57d8344f7b937f818fccdab501, State: Initialized, Role: FOLLOWER
I20260927 22:19:00.072193 8296 consensus_queue.cc:260] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.074090 8296 ts_tablet_manager.cc:1428] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 22:19:00.078584 8295 raft_consensus.cc:348] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.078929 8295 raft_consensus.cc:374] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:00.079048 8295 raft_consensus.cc:729] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df78ad053fce4d7297a7360a5c90c03b, State: Initialized, Role: FOLLOWER
I20260927 22:19:00.079293 8297 raft_consensus.cc:348] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.079619 8297 raft_consensus.cc:374] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:00.079728 8297 raft_consensus.cc:729] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafe4dd559b64f9098b2be14bfdbf381, State: Initialized, Role: FOLLOWER
I20260927 22:19:00.079666 8295 consensus_queue.cc:260] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.080248 8297 consensus_queue.cc:260] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.080780 8148 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:19:00.081252 8279 heartbeater.cc:503] Master 127.23.173.126:41723 was elected leader, sending a full tablet report...
I20260927 22:19:00.081657 8297 ts_tablet_manager.cc:1428] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260927 22:19:00.082741 8295 ts_tablet_manager.cc:1428] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260927 22:19:00.181581 8280 tablet.cc:2369] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:00.208189 8149 tablet.cc:2369] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:00.258816 8017 tablet.cc:2369] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:00.417423 8303 raft_consensus.cc:482] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:00.417758 8303 raft_consensus.cc:504] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.419368 8303 leader_election.cc:288] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445), bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:19:00.425675 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.419625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:38445 (local address 127.23.173.67:44925)
0927 22:19:00.419820 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:00.419840 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:00.419961 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:00.420515 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:00.420518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:00.420544 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:00.420754 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.420764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.422096 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.422102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.423572 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.423584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.424891 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.424897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.425091 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.425126 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:19:00.425246 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:19:00.425375 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:19:00.425673 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.419743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:44925 (local address 127.23.173.66:38445)
0927 22:19:00.419951 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.419957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.419980 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:19:00.420138 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.420159 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.420239 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.420333 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:00.420899 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.421901 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.423736 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.424744 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.424903 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.424986 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.425292 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.425384 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:19:00.426309 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.420235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:58395 (local address 127.23.173.65:40301)
0927 22:19:00.420430 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.420435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.420662 (+ 227us) server_negotiation.cc:407] Connection header received
0927 22:19:00.420774 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.420778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.420858 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.420943 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:00.421558 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.422526 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.424377 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.425352 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.425501 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.425572 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.425941 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.426091 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:19:00.426813 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.419992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:40301 (local address 127.23.173.67:58395)
0927 22:19:00.420555 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:00.420569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:00.420666 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:00.420984 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:00.420991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:00.421016 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:00.421414 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.421426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.422655 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.422658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.424224 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:00.424238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.425547 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.425554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:00.425742 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.425779 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:19:00.426489 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 22:19:00.426594 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":311,"threads_started":1}
I20260927 22:19:00.426668 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c07d1954ec4c27aec682d99778bb5a" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78ad053fce4d7297a7360a5c90c03b" is_pre_election: true
I20260927 22:19:00.427178 8103 raft_consensus.cc:2454] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 0.
I20260927 22:19:00.427847 8168 leader_election.cc:302] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df78ad053fce4d7297a7360a5c90c03b, fafe4dd559b64f9098b2be14bfdbf381; no voters:
I20260927 22:19:00.427748 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c07d1954ec4c27aec682d99778bb5a" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8a7e57d8344f7b937f818fccdab501" is_pre_election: true
I20260927 22:19:00.428263 8303 raft_consensus.cc:2781] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:00.428299 7971 raft_consensus.cc:2454] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 0.
I20260927 22:19:00.428411 8303 raft_consensus.cc:482] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:00.428540 8303 raft_consensus.cc:3037] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:00.430763 8303 raft_consensus.cc:504] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.431305 8303 leader_election.cc:288] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 election: Requested vote from peers df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445), bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301)
I20260927 22:19:00.431792 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c07d1954ec4c27aec682d99778bb5a" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df78ad053fce4d7297a7360a5c90c03b"
I20260927 22:19:00.431846 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c07d1954ec4c27aec682d99778bb5a" candidate_uuid: "fafe4dd559b64f9098b2be14bfdbf381" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8a7e57d8344f7b937f818fccdab501"
I20260927 22:19:00.432024 7971 raft_consensus.cc:3037] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:00.432016 8103 raft_consensus.cc:3037] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:00.435289 7971 raft_consensus.cc:2454] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 1.
I20260927 22:19:00.435289 8103 raft_consensus.cc:2454] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fafe4dd559b64f9098b2be14bfdbf381 in term 1.
I20260927 22:19:00.435693 8168 leader_election.cc:302] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df78ad053fce4d7297a7360a5c90c03b, fafe4dd559b64f9098b2be14bfdbf381; no voters:
I20260927 22:19:00.435949 8303 raft_consensus.cc:2781] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:00.436390 8303 raft_consensus.cc:686] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 1 LEADER]: Becoming Leader. State: Replica: fafe4dd559b64f9098b2be14bfdbf381, State: Running, Role: LEADER
I20260927 22:19:00.436879 8303 consensus_queue.cc:237] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } }
I20260927 22:19:00.441239 7831 catalog_manager.cc:5495] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 reported cstate change: term changed from 0 to 1, leader changed from <none> to fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "fafe4dd559b64f9098b2be14bfdbf381" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fafe4dd559b64f9098b2be14bfdbf381" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:00.485599 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.479964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.23.173.67:46091)
0927 22:19:00.480195 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.480204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.480227 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:19:00.480325 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.480329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.480402 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.480564 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:00.480946 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.481901 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.483509 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.483911 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.484045 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.484544 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:00.485021 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:00.485194 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.485272 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.485347 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:19:00.493875 8103 raft_consensus.cc:1261] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Refusing update from remote peer fafe4dd559b64f9098b2be14bfdbf381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:00.493875 7971 raft_consensus.cc:1261] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Refusing update from remote peer fafe4dd559b64f9098b2be14bfdbf381: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:00.494915 8305 consensus_queue.cc:1035] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8a7e57d8344f7b937f818fccdab501" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 40301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:00.495266 8303 consensus_queue.cc:1035] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df78ad053fce4d7297a7360a5c90c03b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:00.506697 8307 mvcc.cc:205] Tried to move back new op lower bound from 7334082725852119040 to 7334082725633249280. Current Snapshot: MvccSnapshot[applied={T|T < 7334082725852119040}]
I20260927 22:19:00.510149 8308 mvcc.cc:205] Tried to move back new op lower bound from 7334082725852119040 to 7334082725633249280. Current Snapshot: MvccSnapshot[applied={T|T < 7334082725852119040}]
I20260927 22:19:00.521260 7831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:00.521508 7831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:00.524901 7831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Sending DeleteTablet for 3 replicas of tablet 73c07d1954ec4c27aec682d99778bb5a
I20260927 22:19:00.526346 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c07d1954ec4c27aec682d99778bb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:39630
I20260927 22:19:00.526521 7951 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c07d1954ec4c27aec682d99778bb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:52424
I20260927 22:19:00.527835 8214 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c07d1954ec4c27aec682d99778bb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:00 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:44568
I20260927 22:19:00.530408 8317 tablet_replica.cc:330] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: stopping tablet replica
I20260927 22:19:00.530519 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7868
I20260927 22:19:00.530833 8317 raft_consensus.cc:2229] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:00.531303 8317 raft_consensus.cc:2258] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:00.536733 8318 tablet_replica.cc:330] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: stopping tablet replica
I20260927 22:19:00.537119 8318 raft_consensus.cc:2229] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:00.537732 8318 raft_consensus.cc:2258] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:00.539896 8318 ts_tablet_manager.cc:1905] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:00.540844 8317 ts_tablet_manager.cc:1905] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:00.544004 8316 tablet_replica.cc:330] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: stopping tablet replica
I20260927 22:19:00.544431 8316 raft_consensus.cc:2229] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:00.544947 8316 raft_consensus.cc:2258] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:00.548126 8317 ts_tablet_manager.cc:1918] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:00.548286 8317 log.cc:1196] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/73c07d1954ec4c27aec682d99778bb5a
I20260927 22:19:00.548802 8317 ts_tablet_manager.cc:1939] T 73c07d1954ec4c27aec682d99778bb5a P bd8a7e57d8344f7b937f818fccdab501: Deleting consensus metadata
I20260927 22:19:00.549139 8318 ts_tablet_manager.cc:1918] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:00.549301 8318 log.cc:1196] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/73c07d1954ec4c27aec682d99778bb5a
I20260927 22:19:00.549829 8318 ts_tablet_manager.cc:1939] T 73c07d1954ec4c27aec682d99778bb5a P fafe4dd559b64f9098b2be14bfdbf381: Deleting consensus metadata
I20260927 22:19:00.550823 7816 catalog_manager.cc:4841] TS bd8a7e57d8344f7b937f818fccdab501 (127.23.173.65:40301): tablet 73c07d1954ec4c27aec682d99778bb5a (table test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]) successfully deleted
I20260927 22:19:00.551573 7815 catalog_manager.cc:4841] TS fafe4dd559b64f9098b2be14bfdbf381 (127.23.173.67:46091): tablet 73c07d1954ec4c27aec682d99778bb5a (table test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]) successfully deleted
I20260927 22:19:00.553192 8316 ts_tablet_manager.cc:1905] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:00.559187 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.532113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.23.173.65:40301)
0927 22:19:00.532338 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.532347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.538843 (+ 6496us) server_negotiation.cc:407] Connection header received
0927 22:19:00.538924 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.538933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.539035 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.539159 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:00.545684 (+ 6525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.546905 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.548825 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.549487 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.549630 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.552835 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:00.552846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:00.552859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:00.552888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:00.555846 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.556654 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.556663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.556669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.556857 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.557238 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.557243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.557246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.557776 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:00.558121 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.558437 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.558652 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:19:00.559886 8011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:00.563916 8316 ts_tablet_manager.cc:1918] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:00.564080 8316 log.cc:1196] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/73c07d1954ec4c27aec682d99778bb5a
I20260927 22:19:00.564640 8316 ts_tablet_manager.cc:1939] T 73c07d1954ec4c27aec682d99778bb5a P df78ad053fce4d7297a7360a5c90c03b: Deleting consensus metadata
I20260927 22:19:00.566537 7816 catalog_manager.cc:4841] TS df78ad053fce4d7297a7360a5c90c03b (127.23.173.66:38445): tablet 73c07d1954ec4c27aec682d99778bb5a (table test-table [id=635c9e6c6e8f4f20b7524bffa26dc153]) successfully deleted
I20260927 22:19:00.649507 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7868
I20260927 22:19:00.668184 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8020
I20260927 22:19:00.682479 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.669073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.23.173.66:38445)
0927 22:19:00.669286 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.669292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.669313 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:19:00.669513 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.669517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.669612 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.669720 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:00.671136 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.671855 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.673298 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.673724 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.673860 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.676472 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:00.676487 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:00.676502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:00.676533 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:00.679353 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.680014 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.680021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.680023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.680153 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.680529 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.680536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.680539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.680890 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:00.681200 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.681600 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.681819 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:19:00.683151 8143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:00.775202 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8020
I20260927 22:19:00.792775 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8151
I20260927 22:19:00.812512 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:00.793583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.23.173.67:46091)
0927 22:19:00.793844 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:00.793852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:00.793873 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:19:00.794033 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:00.794038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:00.794145 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:00.794281 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:00.795617 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.796850 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.798305 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:00.799023 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:00.799154 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:00.805387 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:00.805401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:00.805419 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:00.805459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:00.809502 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.810229 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.810240 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.810243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.810450 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:00.810828 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:00.810834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:00.810838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:00.811317 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:00.811666 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:00.811853 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 22:19:00.812078 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:19:00.813408 8274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:00.934813 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8151
I20260927 22:19:00.955814 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 7798
I20260927 22:19:00.956933 7861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:01.069458 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 7798
I20260927 22:19:21.091187 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--webserver_interface=127.23.173.126
--webserver_port=45617
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:21.421497 8324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:21.421828 8324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:21.422008 8324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:21.429141 8324 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:19:21.429279 8324 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:19:21.429373 8324 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:19:21.429457 8324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:21.429536 8324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:21.429637 8324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:21.429723 8324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:21.429807 8324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:21.439463 8324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=45617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:21.440754 8324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:21.442091 8324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:21.454334 8332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:21.454308 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:21.454581 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:21.455323 8324 server_base.cc:1034] running on GCE node
I20260927 22:19:21.456039 8324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:21.457347 8324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:21.458567 8324 hybrid_clock.cc:648] HybridClock initialized: now 1790547561458511 us; error 26 us; skew 500 ppm
I20260927 22:19:21.458932 8324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:21 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547561, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:21.462251 8324 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:19:21.463642 8324 webserver.cc:460] Webserver started at http://127.23.173.126:45617/ using document root <none> and password file <none>
I20260927 22:19:21.464331 8324 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:21.464417 8324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:21.469759 8324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:21.472621 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:21.474200 8324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:21.474370 8324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c557d0b9625744e48c9ae3ad4fddf023"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "87a8d92ce4cd2d1d2b32f0c82d39e874"
server_key_iv: "b0717db577e7c8e03e232db468b88a2f"
server_key_version: "encryptionkey@0"
I20260927 22:19:21.475306 8324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:21.487478 8324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:21.491667 8324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:21.491958 8324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:21.510213 8324 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:41723
I20260927 22:19:21.510269 8390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:41723 every 8 connection(s)
I20260927 22:19:21.511742 8324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:21.514233 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8324
I20260927 22:19:21.515066 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:40301
--local_ip_for_outbound_sockets=127.23.173.65
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=45309
--webserver_interface=127.23.173.65
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:19:21.518916 8391 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:19:21.522297 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap starting.
I20260927 22:19:21.552340 8391 log.cc:824] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:21.572726 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:19:21.573416 8391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Bootstrap complete.
I20260927 22:19:21.588655 8391 raft_consensus.cc:348] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:19:21.589946 8391 raft_consensus.cc:729] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Initialized, Role: FOLLOWER
I20260927 22:19:21.590663 8391 consensus_queue.cc:260] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:19:21.590950 8391 raft_consensus.cc:388] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:21.591074 8391 raft_consensus.cc:482] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:21.591240 8391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:19:21.596657 8391 raft_consensus.cc:504] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:19:21.597208 8391 leader_election.cc:302] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c557d0b9625744e48c9ae3ad4fddf023; no voters:
I20260927 22:19:21.597910 8391 leader_election.cc:288] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:19:21.598099 8395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:19:21.599431 8395 raft_consensus.cc:686] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [term 3 LEADER]: Becoming Leader. State: Replica: c557d0b9625744e48c9ae3ad4fddf023, State: Running, Role: LEADER
I20260927 22:19:21.599903 8391 sys_catalog.cc:564] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:21.599975 8395 consensus_queue.cc:237] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } }
I20260927 22:19:21.602453 8397 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c557d0b9625744e48c9ae3ad4fddf023. Latest consensus state: current_term: 3 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:19:21.602478 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c557d0b9625744e48c9ae3ad4fddf023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c557d0b9625744e48c9ae3ad4fddf023" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41723 } } }
I20260927 22:19:21.602873 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:21.602877 8397 sys_catalog.cc:458] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:21.615336 8401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:21.624716 8401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:21.626297 8401 catalog_manager.cc:1259] Loaded cluster ID: 595a15ac28af47349ccd5a015f35eca3
I20260927 22:19:21.626452 8401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:21.631399 8401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:21.633157 8401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Loaded TSK: 0
I20260927 22:19:21.633311 8401 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c557d0b9625744e48c9ae3ad4fddf023: Loaded TSK: 1
F20260927 22:19:21.634500 8401 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f607e9929ad google::LogMessage::Fail() at ??:0
@ 0x7f607e99588a google::LogMessage::SendToLog() at ??:0
I20260927 22:19:21.639756 8412 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f607e9924de google::LogMessage::Flush() at ??:0
@ 0x7f607e993a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f60826f67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f60826e573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f60943c51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f60943c4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f60944502a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f60944500c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f609449a5e9 std::function<>::operator()() at ??:0
@ 0x7f60943c8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f60943c6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f609444e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f609444e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60815de345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60815e191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f60815e1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60815bceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60848966db start_thread at ??:0
@ 0x7f607d82c61f clone at ??:0
W20260927 22:19:21.915874 8393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:21.916203 8393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:21.916365 8393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:21.924175 8393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:21.924337 8393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:21.924536 8393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:21.934541 8393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:40301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=45309
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:21.935812 8393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:21.937078 8393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:21.949479 8439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:21.949448 8438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:21.950641 8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:21.951447 8393 server_base.cc:1034] running on GCE node
I20260927 22:19:21.952041 8393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:21.952989 8393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:21.954231 8393 hybrid_clock.cc:648] HybridClock initialized: now 1790547561954169 us; error 68 us; skew 500 ppm
I20260927 22:19:21.954562 8393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:21 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547561, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:21.957602 8393 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:19:21.959360 8393 webserver.cc:460] Webserver started at http://127.23.173.65:45309/ using document root <none> and password file <none>
I20260927 22:19:21.960306 8393 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:21.960433 8393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:21.965790 8393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:19:21.968528 8448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:21.969852 8393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:19:21.970052 8393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "bd8a7e57d8344f7b937f818fccdab501"
format_stamp: "Formatted at 2026-09-27 22:18:35 on dist-test-slave-3wc3"
server_key: "96e896248f813ea49afe3c752cce8a83"
server_key_iv: "5f0e245e51a700748a62b2d47bebf3a1"
server_key_version: "encryptionkey@0"
I20260927 22:19:21.971021 8393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:21.990234 8393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:21.994253 8393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:21.994563 8393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:21.995618 8393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:21.997030 8393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:21.997144 8393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:21.997285 8393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:21.997390 8393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.032087 8393 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:40301
I20260927 22:19:22.032177 8560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:40301 every 8 connection(s)
I20260927 22:19:22.033548 8393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:22.035739 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8393
I20260927 22:19:22.036525 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:38445
--local_ip_for_outbound_sockets=127.23.173.66
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=44039
--webserver_interface=127.23.173.66
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:19:22.038962 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.036938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:38961)
0927 22:19:22.037793 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.038109 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
W20260927 22:19:22.039651 8561 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:41723: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
I20260927 22:19:22.040518 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.040188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:52853)
0927 22:19:22.040350 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.040416 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:19:22.041388 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.041024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:34019)
0927 22:19:22.041210 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.041285 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:19:22.041610 8561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:19:22.371158 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:22.371445 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:22.371613 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:22.378931 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:22.379035 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:22.379206 8564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:22.388712 8564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=44039
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:22.389945 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:22.391289 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:22.404613 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:22.404587 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:22.404544 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:22.405390 8564 server_base.cc:1034] running on GCE node
I20260927 22:19:22.406004 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:22.406925 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:22.408124 8564 hybrid_clock.cc:648] HybridClock initialized: now 1790547562408081 us; error 41 us; skew 500 ppm
I20260927 22:19:22.408447 8564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:22 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:22.411584 8564 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:19:22.413074 8564 webserver.cc:460] Webserver started at http://127.23.173.66:44039/ using document root <none> and password file <none>
I20260927 22:19:22.413744 8564 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:22.413877 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:22.419298 8564 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:22.421947 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.423369 8564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:19:22.423539 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "df78ad053fce4d7297a7360a5c90c03b"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "bb2bded494a6fe62c6b6116c7df4c5ad"
server_key_iv: "095bc52d3f43ddd89e46afe6e66b3e9a"
server_key_version: "encryptionkey@0"
I20260927 22:19:22.424467 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:22.441305 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:22.445627 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:22.445919 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:22.447126 8564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:22.448493 8564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:22.448580 8564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.448746 8564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:22.448846 8564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.484156 8564 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:38445
I20260927 22:19:22.484248 8691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:38445 every 8 connection(s)
I20260927 22:19:22.485648 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:22.490921 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.488884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:45503)
0927 22:19:22.489670 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.489922 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":304,"threads_started":1}
W20260927 22:19:22.491672 8692 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:41723: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
I20260927 22:19:22.492467 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.492131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:34481)
0927 22:19:22.492285 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.492358 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:19:22.492981 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8564
I20260927 22:19:22.493412 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.493042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:60791)
0927 22:19:22.493193 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.493293 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:19:22.493670 8692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:19:22.493696 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:46091
--local_ip_for_outbound_sockets=127.23.173.67
--tserver_master_addrs=127.23.173.126:41723
--webserver_port=34135
--webserver_interface=127.23.173.67
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:22.813589 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:22.813930 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:22.814150 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:22.821352 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:22.821444 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:22.821551 8695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:22.831447 8695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:46091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=34135
--tserver_master_addrs=127.23.173.126:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:22.832710 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:22.834007 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:22.848524 8700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:22.848630 8703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:22.848526 8701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:22.849663 8695 server_base.cc:1034] running on GCE node
I20260927 22:19:22.850425 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:22.851392 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:22.852641 8695 hybrid_clock.cc:648] HybridClock initialized: now 1790547562852583 us; error 50 us; skew 500 ppm
I20260927 22:19:22.852993 8695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:22 dist-test-slave-3wc3 krb5kdc[7248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:22.856628 8695 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:19:22.858199 8695 webserver.cc:460] Webserver started at http://127.23.173.67:34135/ using document root <none> and password file <none>
I20260927 22:19:22.858858 8695 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:22.858963 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:22.864809 8695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:19:22.867547 8710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.868873 8695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:19:22.869074 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fafe4dd559b64f9098b2be14bfdbf381"
format_stamp: "Formatted at 2026-09-27 22:18:36 on dist-test-slave-3wc3"
server_key: "36d259c1fbd15f0ca77c988018927052"
server_key_iv: "2f8db75b19aa4200126b7c5b8f2b435b"
server_key_version: "encryptionkey@0"
I20260927 22:19:22.870025 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:22.883592 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:22.887715 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:22.888000 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:22.888979 8695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:22.890339 8695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:22.890436 8695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.890569 8695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:22.890666 8695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:22.924031 8695 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:46091
I20260927 22:19:22.924115 8822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:46091 every 8 connection(s)
I20260927 22:19:22.925482 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:22.926434 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8695
I20260927 22:19:22.930843 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.928910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:42727)
0927 22:19:22.929703 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.929931 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":372,"threads_started":1}
W20260927 22:19:22.931476 8823 heartbeater.cc:644] Failed to heartbeat to 127.23.173.126:41723 (0 consecutive failures): Network error: Failed to ping master at 127.23.173.126:41723: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
I20260927 22:19:22.932334 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.931977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:47773)
0927 22:19:22.932130 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.932207 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:19:22.933113 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:22.932800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:56989)
0927 22:19:22.932934 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:22.933012 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:19:22.933336 8823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:19:23.042874 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:23.042140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:43845)
0927 22:19:23.042555 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:23.042651 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":197,"threads_started":1}
I20260927 22:19:23.495100 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:23.494280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:60999)
0927 22:19:23.494768 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:23.494875 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":232,"threads_started":1}
I20260927 22:19:23.934581 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:23.933793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:39923)
0927 22:19:23.934264 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:23.934360 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260927 22:19:24.044441 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:24.043768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:37375)
0927 22:19:24.044198 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:24.044277 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260927 22:19:24.496624 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:24.495972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:59783)
0927 22:19:24.496380 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:24.496463 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":198,"threads_started":1}
I20260927 22:19:24.936256 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:24.935519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:59299)
0927 22:19:24.936005 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:24.936088 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":254,"threads_started":1}
I20260927 22:19:25.046083 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:25.045294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:42305)
0927 22:19:25.045812 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:25.045897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20260927 22:19:25.498167 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:25.497465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:40757)
0927 22:19:25.497869 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:25.497946 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260927 22:19:25.937906 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:25.937231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:54939)
0927 22:19:25.937660 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:25.937741 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260927 22:19:26.047737 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:26.046928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:36005)
0927 22:19:26.047433 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:26.047522 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":235,"threads_started":1}
I20260927 22:19:26.499675 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:26.499004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:49053)
0927 22:19:26.499412 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:26.499490 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20260927 22:19:26.939528 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:26.938773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:39733)
0927 22:19:26.939250 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:26.939336 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":230,"threads_started":1}
I20260927 22:19:27.049417 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:27.048605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:58049)
0927 22:19:27.049066 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:27.049145 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260927 22:19:27.501410 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:27.500630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:37685)
0927 22:19:27.501114 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:27.501239 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260927 22:19:27.941274 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:27.940465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:50859)
0927 22:19:27.940977 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:27.941063 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260927 22:19:28.051141 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:28.050379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:58483)
0927 22:19:28.050831 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:28.050929 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260927 22:19:28.503012 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:28.502295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:53309)
0927 22:19:28.502753 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:28.502848 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260927 22:19:28.942833 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:28.942131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:37141)
0927 22:19:28.942580 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:28.942673 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260927 22:19:29.052865 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:29.052085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:45919)
0927 22:19:29.052565 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:29.052643 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":216,"threads_started":1}
I20260927 22:19:29.504673 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:29.503927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:59935)
0927 22:19:29.504386 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:29.504465 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":197,"threads_started":1}
I20260927 22:19:29.944581 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:29.943740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:48069)
0927 22:19:29.944273 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:29.944359 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":249,"threads_started":1}
I20260927 22:19:30.054741 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:30.053652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:39787)
0927 22:19:30.054397 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:30.054505 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":498,"threads_started":1}
I20260927 22:19:30.506397 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:30.505619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:60707)
0927 22:19:30.506120 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:30.506202 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":245,"threads_started":1}
I20260927 22:19:30.946290 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:30.945493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:49105)
0927 22:19:30.946048 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:30.946128 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260927 22:19:31.056346 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:31.055614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:48423)
0927 22:19:31.056092 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:31.056169 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260927 22:19:31.508098 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:31.507376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:55817)
0927 22:19:31.507843 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:31.507924 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":214,"threads_started":1}
I20260927 22:19:31.947911 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:31.947178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:54189)
0927 22:19:31.947656 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:31.947738 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":238,"threads_started":1}
I20260927 22:19:32.058292 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:32.057320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:55879)
0927 22:19:32.057912 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:32.058040 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":315,"threads_started":1}
I20260927 22:19:32.509909 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:32.509085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:53557)
0927 22:19:32.509627 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:32.509730 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":316,"threads_started":1}
I20260927 22:19:32.949736 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:32.948931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:59535)
0927 22:19:32.949446 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:32.949537 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
I20260927 22:19:33.060060 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:33.059333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:59037)
0927 22:19:33.059779 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:33.059878 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260927 22:19:33.511780 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:33.510963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:55955)
0927 22:19:33.511484 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:33.511590 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":227,"threads_started":1}
I20260927 22:19:33.951673 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:33.950856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:40833)
0927 22:19:33.951384 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:33.951489 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":271,"threads_started":1}
I20260927 22:19:34.062242 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:34.061202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:33413)
0927 22:19:34.061864 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:34.062014 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":310,"threads_started":1}
I20260927 22:19:34.513661 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:34.512834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:55263)
0927 22:19:34.513365 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:34.513490 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":283,"threads_started":1}
I20260927 22:19:34.953540 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:34.952714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:54901)
0927 22:19:34.953291 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:34.953381 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":310,"threads_started":1}
I20260927 22:19:35.063952 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:35.063158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:53945)
0927 22:19:35.063656 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:35.063749 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":266,"threads_started":1}
I20260927 22:19:35.515487 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:35.514676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:33713)
0927 22:19:35.515179 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:35.515285 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260927 22:19:35.955433 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:35.954672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:48617)
0927 22:19:35.955149 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:35.955240 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
I20260927 22:19:36.065724 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:36.064899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:40675)
0927 22:19:36.065434 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:36.065547 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":286,"threads_started":1}
I20260927 22:19:36.517233 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:36.516418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:41561)
0927 22:19:36.516939 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:36.517031 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260927 22:19:36.957098 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:36.956321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:44057)
0927 22:19:36.956819 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:36.956906 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":232,"threads_started":1}
I20260927 22:19:37.067731 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:37.066945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.65:40569)
0927 22:19:37.067429 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:37.067532 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260927 22:19:37.519650 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:37.518628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.66:55441)
0927 22:19:37.519324 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:37.519451 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":401,"threads_started":1}
I20260927 22:19:37.929845 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8393
I20260927 22:19:37.954696 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:37.931057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.23.173.65:40301)
0927 22:19:37.931747 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:37.931772 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:37.931826 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:19:37.933562 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:37.933571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:37.934083 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:37.934413 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:37.936262 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:37.937621 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:37.938908 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:37.939583 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:37.939719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:37.948870 (+ 9151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:37.948879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:37.948895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:37.948933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:37.951824 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:37.952449 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:37.952455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:37.952461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:37.952620 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:37.952947 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:37.952950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:37.952952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:37.953460 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:37.953689 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:37.954087 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 22:19:37.954419 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":280,"threads_started":1}
I20260927 22:19:37.956081 8556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:37.959544 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:37.958545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41723 (local address 127.23.173.67:56241)
0927 22:19:37.959118 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:37.959263 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.126:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
I20260927 22:19:38.043274 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8393
I20260927 22:19:38.061910 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8564
I20260927 22:19:38.078969 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:38.062993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.23.173.66:38445)
0927 22:19:38.063560 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:38.063573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:38.063609 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:19:38.065170 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:38.065179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:38.065609 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:38.065908 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:38.067172 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:38.068450 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:38.069682 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:38.070428 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:38.070556 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:38.073032 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:38.073039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:38.073053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:38.073089 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:38.076016 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:38.076614 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:38.076620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:38.076627 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:38.076794 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:38.077116 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:38.077119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:38.077121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:38.077613 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:38.077859 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:38.078259 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 22:19:38.078664 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260927 22:19:38.080319 8687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:38.165786 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8564
I20260927 22:19:38.183640 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8695
I20260927 22:19:38.202239 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:38.184893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.23.173.67:46091)
0927 22:19:38.185534 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:38.185549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:38.185591 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:19:38.187306 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:38.187317 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:38.187750 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:38.188048 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:38.189359 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:38.190855 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:38.192184 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:38.192997 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:38.193156 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:38.195904 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:38.195912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:38.195928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:38.195965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:38.199081 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:38.199708 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:38.199714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:38.199720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:38.199902 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:38.200228 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:38.200231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:38.200233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:38.200755 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:38.201006 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:38.201392 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:19:38.201831 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":332,"threads_started":1}
I20260927 22:19:38.203694 8818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:38.288281 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8695
2026-09-27T22:19:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64682 ms)
[----------] 30 tests from SecurityITest (295004 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:19:38 dist-test-slave-3wc3 krb5kdc[8880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:19:38 dist-test-slave-3wc3 krb5kdc[8880](info): set up 2 sockets
Sep 27 22:19:38 dist-test-slave-3wc3 krb5kdc[8880](info): commencing operation
krb5kdc: starting...
W20260927 22:19:39.398209 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:19:39 dist-test-slave-3wc3 krb5kdc[8880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:19:39.651067 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42797
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:39.999318 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:39.999614 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:39.999706 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:40.007090 8893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:40.007210 8893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:40.007257 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:40.007313 8893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:40.007355 8893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:40.017359 8893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:40.018764 8893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:40.020044 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:40.032416 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:40.032542 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:40.032697 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:40.032987 8893 server_base.cc:1034] running on GCE node
I20260927 22:19:40.034097 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:40.035611 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:40.036872 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790547580036817 us; error 43 us; skew 500 ppm
I20260927 22:19:40.037256 8893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:40 dist-test-slave-3wc3 krb5kdc[8880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:40.040529 8893 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:19:40.042084 8893 webserver.cc:460] Webserver started at http://127.23.173.126:33583/ using document root <none> and password file <none>
I20260927 22:19:40.042812 8893 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:40.042958 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:40.043277 8893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:40.046394 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "7144cb55deb445c5b46e08a7450a8c5d"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "3b6d593eecaa3df75ec619933ade0021"
server_key_iv: "c34219c4ad46bfd7b502f42ed251cb96"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.047135 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "7144cb55deb445c5b46e08a7450a8c5d"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "3b6d593eecaa3df75ec619933ade0021"
server_key_iv: "c34219c4ad46bfd7b502f42ed251cb96"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.053006 8893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:19:40.057734 8908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:40.059249 8893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:19:40.059423 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "7144cb55deb445c5b46e08a7450a8c5d"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "3b6d593eecaa3df75ec619933ade0021"
server_key_iv: "c34219c4ad46bfd7b502f42ed251cb96"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.059610 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:40.083065 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:40.087392 8893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:40.087680 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:40.106706 8893 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42797
I20260927 22:19:40.106714 8959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42797 every 8 connection(s)
I20260927 22:19:40.108067 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:40.111840 8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:40.114912 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8893
I20260927 22:19:40.115165 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:40.115876 24245 external_mini_cluster.cc:1428] Setting key 11477314c68017dd74ec33b910f42a0b
I20260927 22:19:40.125191 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: Bootstrap starting.
I20260927 22:19:40.128994 8960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:40.130077 8960 log.cc:824] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:19:40 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:40.133361 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: No bootstrap required, opened a new log
I20260927 22:19:40.140583 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:40.118160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.23.173.126:42797)
0927 22:19:40.118928 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:40.118944 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:40.118995 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:19:40.120924 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:40.120956 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:40.121508 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:40.122010 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:40.123491 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.125456 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.126946 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.128030 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.128208 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:40.132593 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:40.132605 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:40.132625 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:40.132676 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:40.136283 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:40.137067 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:40.137074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:40.137080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:40.137252 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:40.137668 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:40.137675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:40.137677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:40.138303 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:40.138623 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:40.139077 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:19:40.139517 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":300,"threads_started":1}
I20260927 22:19:40.141667 8960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } }
I20260927 22:19:40.142112 8960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:40.142241 8960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7144cb55deb445c5b46e08a7450a8c5d, State: Initialized, Role: FOLLOWER
I20260927 22:19:40.142901 8960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } }
I20260927 22:19:40.143158 8960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:40.143319 8960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:40.143525 8960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:40.145754 8960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } }
I20260927 22:19:40.146376 8960 leader_election.cc:302] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7144cb55deb445c5b46e08a7450a8c5d; no voters:
I20260927 22:19:40.146972 8960 leader_election.cc:288] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:40.147100 8965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:40.147799 8965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [term 1 LEADER]: Becoming Leader. State: Replica: 7144cb55deb445c5b46e08a7450a8c5d, State: Running, Role: LEADER
I20260927 22:19:40.148404 8965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } }
I20260927 22:19:40.150067 8960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:40.151427 8966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7144cb55deb445c5b46e08a7450a8c5d. Latest consensus state: current_term: 1 leader_uuid: "7144cb55deb445c5b46e08a7450a8c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } } }
I20260927 22:19:40.152134 8966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:40.151639 8967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7144cb55deb445c5b46e08a7450a8c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7144cb55deb445c5b46e08a7450a8c5d" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42797 } } }
I20260927 22:19:40.152364 8967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d [sys.catalog]: This master's current role is: LEADER
W20260927 22:19:40.165773 8981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:19:40.165922 8981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:19:40.166074 8970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:40.168298 8970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:40.177767 8970 catalog_manager.cc:1347] Generated new cluster ID: ff277fd8bdec4a33addecd923b045ade
I20260927 22:19:40.177996 8970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:40.191506 8970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:40.192886 8970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:40.199545 8970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: Generated new TSK 0
I20260927 22:19:40.200291 8970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:19:40.262142 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:40.594429 8986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:40.594748 8986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:40.594926 8986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:40.602173 8986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:40.602314 8986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:40.602479 8986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:40.612196 8986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:40.613513 8986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:40.614843 8986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:40.628293 8994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:40.628311 8992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:40.628221 8991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:40.629225 8986 server_base.cc:1034] running on GCE node
I20260927 22:19:40.629878 8986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:40.630877 8986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:40.632113 8986 hybrid_clock.cc:648] HybridClock initialized: now 1790547580632070 us; error 49 us; skew 500 ppm
I20260927 22:19:40.632460 8986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:40 dist-test-slave-3wc3 krb5kdc[8880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:40.635931 8986 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:19:40.637398 8986 webserver.cc:460] Webserver started at http://127.23.173.65:33671/ using document root <none> and password file <none>
I20260927 22:19:40.638150 8986 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:40.638298 8986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:40.638612 8986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:40.641909 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "6fc30fa068b24473925ab97d5b197d72"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "43a3850b993c43a5cb1e7470e70f2090"
server_key_iv: "5719b40c7512adb65872d41faa33abd9"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.642773 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "6fc30fa068b24473925ab97d5b197d72"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "43a3850b993c43a5cb1e7470e70f2090"
server_key_iv: "5719b40c7512adb65872d41faa33abd9"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.647878 8986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:19:40.651372 9001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:40.652734 8986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:40.652910 8986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "6fc30fa068b24473925ab97d5b197d72"
format_stamp: "Formatted at 2026-09-27 22:19:40 on dist-test-slave-3wc3"
server_key: "43a3850b993c43a5cb1e7470e70f2090"
server_key_iv: "5719b40c7512adb65872d41faa33abd9"
server_key_version: "encryptionkey@0"
I20260927 22:19:40.653116 8986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:40.669754 8986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:40.673800 8986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:40.674165 8986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:40.675100 8986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:40.676455 8986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:40.676558 8986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:40.676712 8986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:40.676795 8986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:40.709358 8986 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:33781
I20260927 22:19:40.709408 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:33781 every 8 connection(s)
I20260927 22:19:40.710701 8986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:40.718973 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 8986
I20260927 22:19:40.719199 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:40.720029 24245 external_mini_cluster.cc:1428] Setting key 6989af21b316698fe1345e5acd250aba
Sep 27 22:19:40 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:40.731874 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:40.713340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:40661 (local address 127.23.173.126:42797)
0927 22:19:40.713782 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:40.713788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:40.714917 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 22:19:40.716713 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:40.716718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:40.716787 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:40.716885 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:40.718469 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.719299 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.721352 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.721804 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.721926 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:40.726060 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:40.726065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:40.726069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:40.726101 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:40.728724 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:40.729460 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:40.729469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:40.729472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:40.729588 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:40.730086 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:40.730091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:40.730095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:40.730381 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:40.730540 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:40.731262 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 22:19:40.731517 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":204,"threads_started":1}
I20260927 22:19:40.732864 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:40.713950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42797 (local address 127.23.173.65:40661)
0927 22:19:40.714749 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:40.714787 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:40.716444 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:40.717092 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:40.717100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:40.717649 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:40.718289 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:40.718306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.719457 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.719463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:40.721198 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:40.721214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:40.722144 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:40.722150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:40.722350 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:40.723034 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:40.723057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:40.725761 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:40.728875 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:40.728884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:40.728900 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:40.729328 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:40.729700 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:40.729706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:40.729710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:40.729914 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:40.730513 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:40.730521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:40.730880 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:19:40.731148 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 22:19:40.731684 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":309,"threads_started":1}
I20260927 22:19:40.734472 9114 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42797
I20260927 22:19:40.734786 9114 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:40.735526 9114 heartbeater.cc:511] Master 127.23.173.126:42797 requested a full tablet report, sending...
I20260927 22:19:40.737979 8925 ts_manager.cc:194] Registered new tserver with Master: 6fc30fa068b24473925ab97d5b197d72 (127.23.173.65:33781)
I20260927 22:19:40.740216 8925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:40661
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:19:40.780731 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:41.110828 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:41.111147 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:41.111295 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:41.121049 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:41.121160 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:41.121272 9120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:41.130975 9120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:41.132222 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:41.134116 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:41.147264 9125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:41.147466 9126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:41.147342 9128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:41.148396 9120 server_base.cc:1034] running on GCE node
I20260927 22:19:41.148953 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:41.149859 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:41.151122 9120 hybrid_clock.cc:648] HybridClock initialized: now 1790547581151052 us; error 58 us; skew 500 ppm
I20260927 22:19:41.151464 9120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:41 dist-test-slave-3wc3 krb5kdc[8880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:41.154682 9120 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:19:41.156517 9120 webserver.cc:460] Webserver started at http://127.23.173.66:38235/ using document root <none> and password file <none>
I20260927 22:19:41.157230 9120 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:41.157428 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:41.157766 9120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:41.160769 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "bc096287df2543bbbc95f1c6903233fe"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "aab6cf91f1026a24d7bfee622699851c"
server_key_iv: "45e7e1da8ad4745320e659d5ace26b6d"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.161471 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "bc096287df2543bbbc95f1c6903233fe"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "aab6cf91f1026a24d7bfee622699851c"
server_key_iv: "45e7e1da8ad4745320e659d5ace26b6d"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.166839 9120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:19:41.170621 9135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.172068 9120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:41.172255 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "bc096287df2543bbbc95f1c6903233fe"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "aab6cf91f1026a24d7bfee622699851c"
server_key_iv: "45e7e1da8ad4745320e659d5ace26b6d"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.172479 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:41.190737 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:41.195379 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:41.195686 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:41.196660 9120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:41.198040 9120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:41.198117 9120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.198294 9120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:41.198351 9120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.232890 9120 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:34723
I20260927 22:19:41.232949 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:34723 every 8 connection(s)
I20260927 22:19:41.234531 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:41.244604 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9120
I20260927 22:19:41.244801 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:41.245260 24245 external_mini_cluster.cc:1428] Setting key 809ce5bbdb28400efd95c4480cb3af36
Sep 27 22:19:41 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:41.256161 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.237364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:57405 (local address 127.23.173.126:42797)
0927 22:19:41.237818 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.237823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.239082 (+ 1259us) server_negotiation.cc:407] Connection header received
0927 22:19:41.240912 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.240917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.241005 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.241122 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:41.242705 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.243427 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.245032 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.245501 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.245630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.249794 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:41.249799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:41.249803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:41.249835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:41.252685 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.253650 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.253654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.253656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.253780 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.254354 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.254358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.254360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.254629 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:41.254749 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.255617 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.255842 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260927 22:19:41.257519 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.238097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42797 (local address 127.23.173.66:57405)
0927 22:19:41.238910 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.238946 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.240653 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.241402 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.241410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.241915 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:41.242533 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.242555 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.243577 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.243581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.244892 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.244907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.245724 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.245731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.246008 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.246712 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:41.246746 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:41.249538 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:41.252832 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:41.252844 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:41.252864 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:41.253520 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:41.253892 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:41.253900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:41.253906 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:41.254197 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:41.254775 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:41.254786 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:41.255233 (+ 447us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.255538 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.256139 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":379,"threads_started":1}
I20260927 22:19:41.258934 9248 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42797
I20260927 22:19:41.259246 9248 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:41.260085 9248 heartbeater.cc:511] Master 127.23.173.126:42797 requested a full tablet report, sending...
I20260927 22:19:41.261622 8925 ts_manager.cc:194] Registered new tserver with Master: bc096287df2543bbbc95f1c6903233fe (127.23.173.66:34723)
I20260927 22:19:41.262521 8925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:57405
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:19:41.305527 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:41.632789 9254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:41.633086 9254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:41.633248 9254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:41.640494 9254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:41.640630 9254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:41.640808 9254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:41.650419 9254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:41.651746 9254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:41.653033 9254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:41.666698 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:41.666760 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:41.667153 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:41.667660 9254 server_base.cc:1034] running on GCE node
I20260927 22:19:41.668272 9254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:41.669214 9254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:41.670437 9254 hybrid_clock.cc:648] HybridClock initialized: now 1790547581670401 us; error 58 us; skew 500 ppm
I20260927 22:19:41.670745 9254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:41 dist-test-slave-3wc3 krb5kdc[8880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:41.673812 9254 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:19:41.675302 9254 webserver.cc:460] Webserver started at http://127.23.173.67:43691/ using document root <none> and password file <none>
I20260927 22:19:41.675976 9254 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:41.676120 9254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:41.676427 9254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:41.679507 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "fe3f46d981ee46cfab64b67905af6ee2"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "08338ec84558d83de9bd351712290114"
server_key_iv: "8cd5d8645a041756706ec5afbd3cf153"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.680280 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "fe3f46d981ee46cfab64b67905af6ee2"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "08338ec84558d83de9bd351712290114"
server_key_iv: "8cd5d8645a041756706ec5afbd3cf153"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.685253 9254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:19:41.688724 9269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.690207 9254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:41.690442 9254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "fe3f46d981ee46cfab64b67905af6ee2"
format_stamp: "Formatted at 2026-09-27 22:19:41 on dist-test-slave-3wc3"
server_key: "08338ec84558d83de9bd351712290114"
server_key_iv: "8cd5d8645a041756706ec5afbd3cf153"
server_key_version: "encryptionkey@0"
I20260927 22:19:41.690659 9254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:41.722909 9254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:41.727227 9254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:41.727547 9254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:41.728591 9254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:41.730026 9254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:41.730129 9254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.730294 9254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:41.730369 9254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:41.743570 9114 heartbeater.cc:503] Master 127.23.173.126:42797 was elected leader, sending a full tablet report...
I20260927 22:19:41.764978 9254 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:45971
I20260927 22:19:41.765020 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:45971 every 8 connection(s)
I20260927 22:19:41.766374 9254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:41.769769 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9254
I20260927 22:19:41.769986 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:41.770464 24245 external_mini_cluster.cc:1428] Setting key 2219a4e26f72f217c3971f3d38032b3e
Sep 27 22:19:41 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:41.786954 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.768952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:38509 (local address 127.23.173.126:42797)
0927 22:19:41.769500 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.769507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.770679 (+ 1172us) server_negotiation.cc:407] Connection header received
0927 22:19:41.772541 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.772547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.772642 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.772734 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:41.774502 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.775419 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.777033 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.777468 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.777572 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.780915 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:41.780928 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:41.780931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:41.780963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:41.783655 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.784455 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.784462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.784467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.784596 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.785147 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.785152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.785155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.785465 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:41.785634 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.786360 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.786576 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260927 22:19:41.787884 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.769599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:42797 (local address 127.23.173.67:38509)
0927 22:19:41.770481 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.770539 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.772299 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.773021 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.773034 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.773701 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:41.774340 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.774357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.775578 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.775582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.776905 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.776917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.777672 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.777677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.777871 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.778574 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:41.778599 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:41.780720 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:41.783796 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:41.783803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:41.783815 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:41.784284 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:41.784738 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:41.784742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:41.784744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:41.785010 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:41.785609 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:41.785618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:41.785940 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.786257 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.786798 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":437,"threads_started":1}
I20260927 22:19:41.789341 9382 heartbeater.cc:347] Connected to a master server at 127.23.173.126:42797
I20260927 22:19:41.789590 9382 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:41.790298 9382 heartbeater.cc:511] Master 127.23.173.126:42797 requested a full tablet report, sending...
I20260927 22:19:41.791603 8925 ts_manager.cc:194] Registered new tserver with Master: fe3f46d981ee46cfab64b67905af6ee2 (127.23.173.67:45971)
I20260927 22:19:41.792371 8925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:38509
I20260927 22:19:41.798897 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:41.816972 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.803594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.23.173.126:42797)
0927 22:19:41.803853 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.803860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.804225 (+ 365us) server_negotiation.cc:407] Connection header received
0927 22:19:41.804363 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.804367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.804425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.804508 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:41.806282 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.807013 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.808498 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.808893 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.808984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.811665 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:41.811671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:41.811674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:41.811708 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:41.814445 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.815066 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.815069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.815072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.815205 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:41.815522 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:41.815525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:41.815527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:41.815869 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:41.816055 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.816415 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.816596 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:19:41.821141 8925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:41.828603 8925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:41.852825 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.840960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33781 (local address 127.0.0.1:60416)
0927 22:19:41.843336 (+ 2376us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.843378 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.843589 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.844424 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.844432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.844460 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:41.845192 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.845205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.847393 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.847398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.849860 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.849876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.851830 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.851837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.852040 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.852089 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.852248 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.852435 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2100,"thread_start_us":2564,"threads_started":1}
I20260927 22:19:41.853886 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.843916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34723 (local address 127.0.0.1:46608)
0927 22:19:41.844492 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.844518 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.844682 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.846647 (+ 1965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.846651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.846677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:41.847043 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.847055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.848847 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.848852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.850845 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.850859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.852989 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.852997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.853163 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.853205 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.853299 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.853453 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"spinlock_wait_cycles":1408,"thread_start_us":257,"threads_started":1}
I20260927 22:19:41.853930 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.842004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.23.173.65:33781)
0927 22:19:41.843378 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.843389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.843770 (+ 381us) server_negotiation.cc:407] Connection header received
0927 22:19:41.843894 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.843903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.844204 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.844424 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:41.845389 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.847191 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.850080 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.851632 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.851779 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.852017 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.852314 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.853579 (+ 1265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":730,"threads_started":1}
I20260927 22:19:41.854189 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.844159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.23.173.66:34723)
0927 22:19:41.845102 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.845111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.845139 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:19:41.845230 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.845238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.846208 (+ 970us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.846401 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:41.847225 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.848661 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.851022 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.852505 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.852651 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.852833 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.853639 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.853838 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":609,"threads_started":1}
I20260927 22:19:41.854010 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.841218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:45971 (local address 127.0.0.1:42298)
0927 22:19:41.843816 (+ 2598us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.843833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.844875 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.845828 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.845833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.845859 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:41.846258 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.846281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.849141 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.849151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.851167 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.851180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.853239 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.853246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.853418 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.853449 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.853531 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.853649 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2476,"thread_start_us":1747,"threads_started":1}
I20260927 22:19:41.854641 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.841394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.23.173.67:45971)
0927 22:19:41.842753 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.842765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.843922 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 22:19:41.845073 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.845083 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.845335 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.845651 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:41.846481 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.848142 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.851338 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.852726 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.852867 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.853094 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.854148 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.854322 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":1016,"threads_started":1}
I20260927 22:19:41.857745 9049 tablet_service.cc:1467] Processing CreateTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d (DEFAULT_TABLE table=test-table [id=8773fdbe77714612a4f80c574fd603de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:41.857847 9317 tablet_service.cc:1467] Processing CreateTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d (DEFAULT_TABLE table=test-table [id=8773fdbe77714612a4f80c574fd603de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:41.859380 9317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17e6ae84f62404589cc2adeeaf27c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:41.859426 9049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17e6ae84f62404589cc2adeeaf27c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:41.860034 9183 tablet_service.cc:1467] Processing CreateTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d (DEFAULT_TABLE table=test-table [id=8773fdbe77714612a4f80c574fd603de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:41.861353 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17e6ae84f62404589cc2adeeaf27c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:41.872381 9401 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Bootstrap starting.
I20260927 22:19:41.874068 9402 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Bootstrap starting.
I20260927 22:19:41.875582 9403 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Bootstrap starting.
I20260927 22:19:41.876057 9401 tablet_bootstrap.cc:654] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:41.877390 9401 log.cc:824] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:41.878190 9402 tablet_bootstrap.cc:654] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:41.878789 9403 tablet_bootstrap.cc:654] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:41.879324 9402 log.cc:824] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:41.879827 9403 log.cc:824] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:41.881585 9401 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: No bootstrap required, opened a new log
I20260927 22:19:41.881927 9401 ts_tablet_manager.cc:1397] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 22:19:41.884042 9403 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: No bootstrap required, opened a new log
I20260927 22:19:41.884294 9403 ts_tablet_manager.cc:1397] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 22:19:41.888340 9402 tablet_bootstrap.cc:492] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: No bootstrap required, opened a new log
I20260927 22:19:41.888597 9402 ts_tablet_manager.cc:1397] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260927 22:19:41.891775 9403 raft_consensus.cc:348] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.892256 9403 raft_consensus.cc:374] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:41.892401 9403 raft_consensus.cc:729] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc096287df2543bbbc95f1c6903233fe, State: Initialized, Role: FOLLOWER
I20260927 22:19:41.893015 9403 consensus_queue.cc:260] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.893741 9401 raft_consensus.cc:348] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.894171 9248 heartbeater.cc:503] Master 127.23.173.126:42797 was elected leader, sending a full tablet report...
I20260927 22:19:41.894191 9401 raft_consensus.cc:374] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:41.894274 9401 raft_consensus.cc:729] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3f46d981ee46cfab64b67905af6ee2, State: Initialized, Role: FOLLOWER
I20260927 22:19:41.894840 9403 ts_tablet_manager.cc:1428] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260927 22:19:41.894822 9401 consensus_queue.cc:260] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.895879 9382 heartbeater.cc:503] Master 127.23.173.126:42797 was elected leader, sending a full tablet report...
I20260927 22:19:41.896447 9401 ts_tablet_manager.cc:1428] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 22:19:41.899333 9402 raft_consensus.cc:348] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.899784 9402 raft_consensus.cc:374] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:41.899901 9402 raft_consensus.cc:729] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc30fa068b24473925ab97d5b197d72, State: Initialized, Role: FOLLOWER
I20260927 22:19:41.900530 9402 consensus_queue.cc:260] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.902077 9402 ts_tablet_manager.cc:1428] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 22:19:41.948320 9408 raft_consensus.cc:482] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:41.948640 9408 raft_consensus.cc:504] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.950423 9408 leader_election.cc:288] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc30fa068b24473925ab97d5b197d72 (127.23.173.65:33781), bc096287df2543bbbc95f1c6903233fe (127.23.173.66:34723)
I20260927 22:19:41.956017 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.950638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33781 (local address 127.23.173.67:36233)
0927 22:19:41.950977 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.950999 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.951129 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.951585 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.951590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.951626 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:41.951818 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.951827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.952858 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.952865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.954175 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.954187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.955293 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.955299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.955480 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.955504 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.955604 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.955754 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260927 22:19:41.956017 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.950879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:36233 (local address 127.23.173.65:33781)
0927 22:19:41.951086 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.951091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.951110 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:19:41.951324 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.951333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.951427 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.951525 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:41.951989 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.952718 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.954377 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.955138 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.955322 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.955400 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.955640 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.955765 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:19:41.957130 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.951077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:60961 (local address 127.23.173.66:34723)
0927 22:19:41.951304 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:41.951313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:41.951873 (+ 560us) server_negotiation.cc:407] Connection header received
0927 22:19:41.952081 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:41.952085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:41.952144 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:41.952238 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:41.952780 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.953448 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.955266 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.956154 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.956311 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.956429 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:41.956749 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 22:19:41.956882 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:19:41.957182 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:41.950722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:34723 (local address 127.23.173.67:60961)
0927 22:19:41.951750 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:41.951771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:41.951928 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:41.952251 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:41.952256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:41.952278 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:41.952613 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.952621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.953593 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.953598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.955105 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:41.955116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:41.956471 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:41.956484 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:41.956593 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:41.956614 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:19:41.956718 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:19:41.956867 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":665,"threads_started":1}
I20260927 22:19:41.957173 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17e6ae84f62404589cc2adeeaf27c5d" candidate_uuid: "fe3f46d981ee46cfab64b67905af6ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc30fa068b24473925ab97d5b197d72" is_pre_election: true
I20260927 22:19:41.957942 9069 raft_consensus.cc:2454] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3f46d981ee46cfab64b67905af6ee2 in term 0.
I20260927 22:19:41.958298 9203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17e6ae84f62404589cc2adeeaf27c5d" candidate_uuid: "fe3f46d981ee46cfab64b67905af6ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc096287df2543bbbc95f1c6903233fe" is_pre_election: true
I20260927 22:19:41.958675 9271 leader_election.cc:302] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc30fa068b24473925ab97d5b197d72, fe3f46d981ee46cfab64b67905af6ee2; no voters:
I20260927 22:19:41.958730 9203 raft_consensus.cc:2454] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3f46d981ee46cfab64b67905af6ee2 in term 0.
I20260927 22:19:41.959098 9408 raft_consensus.cc:2781] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:41.959270 9408 raft_consensus.cc:482] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:41.959414 9408 raft_consensus.cc:3037] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:41.961627 9408 raft_consensus.cc:504] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.962339 9408 leader_election.cc:288] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc30fa068b24473925ab97d5b197d72 (127.23.173.65:33781), bc096287df2543bbbc95f1c6903233fe (127.23.173.66:34723)
I20260927 22:19:41.962780 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17e6ae84f62404589cc2adeeaf27c5d" candidate_uuid: "fe3f46d981ee46cfab64b67905af6ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc30fa068b24473925ab97d5b197d72"
I20260927 22:19:41.962844 9203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17e6ae84f62404589cc2adeeaf27c5d" candidate_uuid: "fe3f46d981ee46cfab64b67905af6ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc096287df2543bbbc95f1c6903233fe"
I20260927 22:19:41.963053 9069 raft_consensus.cc:3037] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:41.963040 9203 raft_consensus.cc:3037] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 0 FOLLOWER]: Advancing to term 1
W20260927 22:19:41.963080 9115 tablet.cc:2369] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:41.965438 9203 raft_consensus.cc:2454] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3f46d981ee46cfab64b67905af6ee2 in term 1.
I20260927 22:19:41.965535 9069 raft_consensus.cc:2454] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3f46d981ee46cfab64b67905af6ee2 in term 1.
I20260927 22:19:41.965858 9273 leader_election.cc:302] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc096287df2543bbbc95f1c6903233fe, fe3f46d981ee46cfab64b67905af6ee2; no voters:
I20260927 22:19:41.966189 9408 raft_consensus.cc:2781] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:41.966750 9408 raft_consensus.cc:686] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 1 LEADER]: Becoming Leader. State: Replica: fe3f46d981ee46cfab64b67905af6ee2, State: Running, Role: LEADER
I20260927 22:19:41.967267 9408 consensus_queue.cc:237] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } }
I20260927 22:19:41.971623 8925 catalog_manager.cc:5495] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3f46d981ee46cfab64b67905af6ee2 (127.23.173.67). New cstate: current_term: 1 leader_uuid: "fe3f46d981ee46cfab64b67905af6ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3f46d981ee46cfab64b67905af6ee2" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 45971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 } health_report { overall_health: UNKNOWN } } }
W20260927 22:19:41.986573 9249 tablet.cc:2369] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:42.009145 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:42.003469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.23.173.67:45971)
0927 22:19:42.003690 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:42.003696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:42.003801 (+ 105us) server_negotiation.cc:407] Connection header received
0927 22:19:42.004067 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:42.004071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:42.004147 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:42.004301 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:42.004680 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.005425 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.006962 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.007371 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.007511 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:42.008020 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:42.008565 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:42.008737 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:42.008796 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 22:19:42.008874 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:19:42.017215 9203 raft_consensus.cc:1261] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 1 FOLLOWER]: Refusing update from remote peer fe3f46d981ee46cfab64b67905af6ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:42.017246 9069 raft_consensus.cc:1261] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 1 FOLLOWER]: Refusing update from remote peer fe3f46d981ee46cfab64b67905af6ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 22:19:42.018108 9383 tablet.cc:2369] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:42.018231 9411 consensus_queue.cc:1035] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc30fa068b24473925ab97d5b197d72" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:42.018718 9408 consensus_queue.cc:1035] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc096287df2543bbbc95f1c6903233fe" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:42.030695 9414 mvcc.cc:205] Tried to move back new op lower bound from 7334082895932559360 to 7334082895741976576. Current Snapshot: MvccSnapshot[applied={T|T < 7334082895932559360}]
I20260927 22:19:42.034266 9415 mvcc.cc:205] Tried to move back new op lower bound from 7334082895932559360 to 7334082895741976576. Current Snapshot: MvccSnapshot[applied={T|T < 7334082895932559360}]
I20260927 22:19:42.043298 8925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:42.043550 8925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:42.047168 8925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7144cb55deb445c5b46e08a7450a8c5d: Sending DeleteTablet for 3 replicas of tablet f17e6ae84f62404589cc2adeeaf27c5d
I20260927 22:19:42.048274 9049 tablet_service.cc:1514] Processing DeleteTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:42 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:60416
I20260927 22:19:42.048519 9183 tablet_service.cc:1514] Processing DeleteTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:42 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:46608
I20260927 22:19:42.048904 9317 tablet_service.cc:1514] Processing DeleteTablet for tablet f17e6ae84f62404589cc2adeeaf27c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:42 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:42298
I20260927 22:19:42.049376 9424 tablet_replica.cc:330] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: stopping tablet replica
I20260927 22:19:42.049765 9424 raft_consensus.cc:2229] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:42.050319 9424 raft_consensus.cc:2258] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:42.050839 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8986
I20260927 22:19:42.051105 9423 tablet_replica.cc:330] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: stopping tablet replica
I20260927 22:19:42.051575 9423 raft_consensus.cc:2229] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:42.052019 9423 raft_consensus.cc:2258] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:42.053987 9423 ts_tablet_manager.cc:1905] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:42.061017 9423 ts_tablet_manager.cc:1918] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:42.061157 9423 log.cc:1196] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/f17e6ae84f62404589cc2adeeaf27c5d
I20260927 22:19:42.061674 9423 ts_tablet_manager.cc:1939] T f17e6ae84f62404589cc2adeeaf27c5d P 6fc30fa068b24473925ab97d5b197d72: Deleting consensus metadata
I20260927 22:19:42.063721 8910 catalog_manager.cc:4841] TS 6fc30fa068b24473925ab97d5b197d72 (127.23.173.65:33781): tablet f17e6ae84f62404589cc2adeeaf27c5d (table test-table [id=8773fdbe77714612a4f80c574fd603de]) successfully deleted
I20260927 22:19:42.063938 9424 ts_tablet_manager.cc:1905] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:42.070056 9425 tablet_replica.cc:330] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: stopping tablet replica
I20260927 22:19:42.070509 9425 raft_consensus.cc:2229] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:42.071173 9425 raft_consensus.cc:2258] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:42.072748 9424 ts_tablet_manager.cc:1918] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:42.072933 9424 log.cc:1196] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/f17e6ae84f62404589cc2adeeaf27c5d
I20260927 22:19:42.073350 9425 ts_tablet_manager.cc:1905] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:42.073493 9424 ts_tablet_manager.cc:1939] T f17e6ae84f62404589cc2adeeaf27c5d P bc096287df2543bbbc95f1c6903233fe: Deleting consensus metadata
I20260927 22:19:42.075359 8912 catalog_manager.cc:4841] TS bc096287df2543bbbc95f1c6903233fe (127.23.173.66:34723): tablet f17e6ae84f62404589cc2adeeaf27c5d (table test-table [id=8773fdbe77714612a4f80c574fd603de]) successfully deleted
I20260927 22:19:42.079492 9425 ts_tablet_manager.cc:1918] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:42.079668 9425 log.cc:1196] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/f17e6ae84f62404589cc2adeeaf27c5d
I20260927 22:19:42.080228 9425 ts_tablet_manager.cc:1939] T f17e6ae84f62404589cc2adeeaf27c5d P fe3f46d981ee46cfab64b67905af6ee2: Deleting consensus metadata
Sep 27 22:19:42 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:19:42.081697 8912 catalog_manager.cc:4841] TS fe3f46d981ee46cfab64b67905af6ee2 (127.23.173.67:45971): tablet f17e6ae84f62404589cc2adeeaf27c5d (table test-table [id=8773fdbe77714612a4f80c574fd603de]) successfully deleted
I20260927 22:19:42.088912 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:42.052305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.23.173.65:33781)
0927 22:19:42.052504 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:42.052512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:42.064837 (+ 12325us) server_negotiation.cc:407] Connection header received
0927 22:19:42.065109 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:42.065114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:42.065191 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:42.065263 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:42.067077 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.068194 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.070192 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.070839 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.070936 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:42.082710 (+ 11774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:42.082724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:42.082738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:42.082806 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:42.085705 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.086687 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.086694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.086696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.086843 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.087340 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.087344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.087346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.087691 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:42.087921 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:42.088263 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:19:42.088465 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:19:42.089600 9109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:42.204303 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8986
I20260927 22:19:42.227973 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9120
Sep 27 22:19:42 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:19:42.247681 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:42.228765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.23.173.66:34723)
0927 22:19:42.228966 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:42.228970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:42.228988 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:19:42.229221 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:42.229225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:42.229298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:42.229379 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:42.231017 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.231772 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.233829 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.234599 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.234720 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:42.240318 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:42.240334 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:42.240352 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:42.240402 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:42.244564 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.245370 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.245380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.245384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.245532 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.245910 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.245915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.245919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.246449 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:42.246750 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:42.247079 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:19:42.247306 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:19:42.248391 9243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:42.398566 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9120
I20260927 22:19:42.416376 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9254
Sep 27 22:19:42 dist-test-slave-3wc3 krb5kdc[8880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:19:42.434226 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:42.417299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.23.173.67:45971)
0927 22:19:42.417503 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:42.417509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:42.417532 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:19:42.417608 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:42.417614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:42.417705 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:42.417823 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:42.419258 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.420511 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.421768 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:42.422507 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:42.422639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:42.427019 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:42.427035 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:42.427056 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:42.427102 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:42.431213 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.431913 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.431923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.431927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.432148 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:42.432531 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:42.432537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:42.432542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:42.433001 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:42.433336 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:42.433489 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:19:42.433728 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:19:42.435014 9377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:42.558773 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9254
I20260927 22:19:42.575699 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 8893
I20260927 22:19:42.576862 8955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:42.687999 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 8893
2026-09-27T22:19:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4402 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:42Z Disabled control of system clock
I20260927 22:19:42.764599 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39253
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:43.094095 9434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:43.094376 9434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:43.094514 9434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:43.101691 9434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:43.101795 9434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:43.101895 9434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:43.102025 9434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:43.102078 9434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:43.111105 9434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:39253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:43.112355 9434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:43.114096 9434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:43.125641 9439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:43.125730 9442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:43.125679 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:43.126438 9434 server_base.cc:1034] running on GCE node
I20260927 22:19:43.127149 9434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:43.128554 9434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:43.129794 9434 hybrid_clock.cc:648] HybridClock initialized: now 1790547583129746 us; error 45 us; skew 500 ppm
I20260927 22:19:43.130165 9434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:43.135809 9434 webserver.cc:460] Webserver started at http://127.23.173.126:36969/ using document root <none> and password file <none>
I20260927 22:19:43.136538 9434 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:43.136682 9434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:43.137009 9434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:43.139907 9434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c97c0ebae5524dc48eb1ceeac1683f5b"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "b8806b235b957d4d00c3b307d0ea6f37"
server_key_iv: "3b1bcdb85e5cc8e8a6b493b471471f2d"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.140722 9434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c97c0ebae5524dc48eb1ceeac1683f5b"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "b8806b235b957d4d00c3b307d0ea6f37"
server_key_iv: "3b1bcdb85e5cc8e8a6b493b471471f2d"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.145819 9434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:19:43.149232 9448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:43.150787 9434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:43.150974 9434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c97c0ebae5524dc48eb1ceeac1683f5b"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "b8806b235b957d4d00c3b307d0ea6f37"
server_key_iv: "3b1bcdb85e5cc8e8a6b493b471471f2d"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.151190 9434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:43.171448 9434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:43.172312 9434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:43.172626 9434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:43.191875 9434 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:39253
I20260927 22:19:43.191897 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:39253 every 8 connection(s)
I20260927 22:19:43.193238 9434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:43.196061 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9434
I20260927 22:19:43.196250 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:43.196916 24245 external_mini_cluster.cc:1428] Setting key 92aa410971bf57672ae9992dfac0451d
I20260927 22:19:43.198045 9500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:43.210254 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:43.199073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.23.173.126:39253)
0927 22:19:43.200053 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:43.200064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:43.200105 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:19:43.201616 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:43.201636 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:43.202159 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:43.202581 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:43.203137 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.204471 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.205805 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.206520 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.206714 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:43.207974 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:43.207981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:43.207998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:43.208536 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:43.208614 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:43.208635 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:43.208761 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:43.208991 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:19:43.209286 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":374,"threads_started":1}
I20260927 22:19:43.211423 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: Bootstrap starting.
I20260927 22:19:43.214995 9500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:43.216105 9500 log.cc:824] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:43.219120 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: No bootstrap required, opened a new log
I20260927 22:19:43.226468 9500 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } }
I20260927 22:19:43.226864 9500 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:43.226970 9500 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97c0ebae5524dc48eb1ceeac1683f5b, State: Initialized, Role: FOLLOWER
I20260927 22:19:43.227522 9500 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } }
I20260927 22:19:43.227766 9500 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:43.227913 9500 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:43.228082 9500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:43.230264 9500 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } }
I20260927 22:19:43.230852 9500 leader_election.cc:302] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97c0ebae5524dc48eb1ceeac1683f5b; no voters:
I20260927 22:19:43.231479 9500 leader_election.cc:288] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:43.231647 9505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:43.232709 9505 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [term 1 LEADER]: Becoming Leader. State: Replica: c97c0ebae5524dc48eb1ceeac1683f5b, State: Running, Role: LEADER
I20260927 22:19:43.233206 9505 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } }
I20260927 22:19:43.233462 9500 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:43.236212 9506 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } } }
I20260927 22:19:43.236205 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97c0ebae5524dc48eb1ceeac1683f5b. Latest consensus state: current_term: 1 leader_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97c0ebae5524dc48eb1ceeac1683f5b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 39253 } } }
I20260927 22:19:43.236680 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:43.236678 9506 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:43.238401 9511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:43.243173 9511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:43.251910 9511 catalog_manager.cc:1347] Generated new cluster ID: 8838796db0d54bfcb34a473bdddb62b8
I20260927 22:19:43.252028 9511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:43.281052 9511 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:43.282889 9511 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:43.298647 9511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: Generated new TSK 0
I20260927 22:19:43.299742 9511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:19:43.308367 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--builtin_ntp_servers=127.23.173.84:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:43.644302 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:43.644626 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:43.644791 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:43.652503 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:43.652647 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:43.652814 9524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:43.662374 9524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:43.663651 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:43.665361 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:43.678135 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:43.678198 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:43.678225 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:43.679087 9524 server_base.cc:1034] running on GCE node
I20260927 22:19:43.679718 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:43.680665 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:43.681910 9524 hybrid_clock.cc:648] HybridClock initialized: now 1790547583681841 us; error 58 us; skew 500 ppm
I20260927 22:19:43.682299 9524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:43.684753 9524 webserver.cc:460] Webserver started at http://127.23.173.65:45421/ using document root <none> and password file <none>
I20260927 22:19:43.685487 9524 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:43.685614 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:43.685933 9524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:43.688836 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "8aa4f889d55f47949689e93ce03e54ac"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "de7f186f371e9c3b211d3d9c7cb0a716"
server_key_iv: "46d0b6cf776fe61cbb082094683c7d53"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.689627 9524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "8aa4f889d55f47949689e93ce03e54ac"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "de7f186f371e9c3b211d3d9c7cb0a716"
server_key_iv: "46d0b6cf776fe61cbb082094683c7d53"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.694908 9524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:19:43.698436 9538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:43.699870 9524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:19:43.700066 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "8aa4f889d55f47949689e93ce03e54ac"
format_stamp: "Formatted at 2026-09-27 22:19:43 on dist-test-slave-3wc3"
server_key: "de7f186f371e9c3b211d3d9c7cb0a716"
server_key_iv: "46d0b6cf776fe61cbb082094683c7d53"
server_key_version: "encryptionkey@0"
I20260927 22:19:43.700311 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:43.719519 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:43.720422 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:43.720741 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:43.721709 9524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:43.723259 9524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:43.723428 9524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:43.723568 9524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:43.723663 9524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:43.757583 9524 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:46817
I20260927 22:19:43.757635 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:46817 every 8 connection(s)
I20260927 22:19:43.759019 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:43.762969 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9524
I20260927 22:19:43.763146 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:43.763638 24245 external_mini_cluster.cc:1428] Setting key f45532451d34b6110b3717b6569a8d3c
I20260927 22:19:43.766422 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--builtin_ntp_servers=127.23.173.84:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:43.772013 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:43.761553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:39275 (local address 127.23.173.126:39253)
0927 22:19:43.762003 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:43.762009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:43.763306 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 22:19:43.765401 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:43.765406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:43.765496 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:43.765625 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:43.766851 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.767566 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.769122 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.769552 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.769687 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:43.771161 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:43.771166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:43.771169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:43.771186 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:43.771223 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:43.771235 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:43.771394 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:43.771576 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 22:19:43.771673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
I20260927 22:19:43.773041 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:43.762128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39253 (local address 127.23.173.65:39275)
0927 22:19:43.763098 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:43.763158 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:43.765091 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:43.765873 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:43.765881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:43.766027 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:43.766689 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:43.766708 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.767716 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.767722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:43.768953 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:43.768965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:43.769787 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:43.769794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:43.770097 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:43.770812 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:43.770814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:43.770871 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:43.770953 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:43.771331 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:43.771343 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:43.771351 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:19:43.771554 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 22:19:43.771961 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260927 22:19:43.774745 9651 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39253
I20260927 22:19:43.775053 9651 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:43.775911 9651 heartbeater.cc:511] Master 127.23.173.126:39253 requested a full tablet report, sending...
I20260927 22:19:43.778256 9465 ts_manager.cc:194] Registered new tserver with Master: 8aa4f889d55f47949689e93ce03e54ac (127.23.173.65:46817)
I20260927 22:19:43.780427 9465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.65:39275
W20260927 22:19:44.097937 9655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:44.098256 9655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:44.098342 9655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:44.105640 9655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:44.105727 9655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:44.105829 9655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:44.114403 9655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:44.115533 9655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:44.117415 9655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:44.130249 9660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:44.130353 9661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:44.130421 9663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:44.131073 9655 server_base.cc:1034] running on GCE node
I20260927 22:19:44.131757 9655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:44.132764 9655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:44.134060 9655 hybrid_clock.cc:648] HybridClock initialized: now 1790547584133989 us; error 50 us; skew 500 ppm
I20260927 22:19:44.134416 9655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:44.136739 9655 webserver.cc:460] Webserver started at http://127.23.173.66:43929/ using document root <none> and password file <none>
I20260927 22:19:44.137527 9655 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:44.137686 9655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:44.138098 9655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:44.141131 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "a1984138bf4e41b499ac008e2e32939b"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "faca2bd3aad8e34d7b3ca0574e986331"
server_key_iv: "0dcb61723aa199fe9348622ad95bebe6"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.142051 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "a1984138bf4e41b499ac008e2e32939b"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "faca2bd3aad8e34d7b3ca0574e986331"
server_key_iv: "0dcb61723aa199fe9348622ad95bebe6"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.147600 9655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:19:44.151186 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.152606 9655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:19:44.152788 9655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "a1984138bf4e41b499ac008e2e32939b"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "faca2bd3aad8e34d7b3ca0574e986331"
server_key_iv: "0dcb61723aa199fe9348622ad95bebe6"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.153023 9655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:44.176432 9655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:44.177340 9655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:44.177673 9655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:44.178791 9655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:44.180342 9655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:44.180485 9655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.180632 9655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:44.180729 9655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.220681 9655 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:35243
I20260927 22:19:44.220767 9781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:35243 every 8 connection(s)
I20260927 22:19:44.222038 9655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:44.223771 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9655
I20260927 22:19:44.223949 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:44.224467 24245 external_mini_cluster.cc:1428] Setting key d0e001f980f2c96751168a7d64b2491b
I20260927 22:19:44.227469 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--builtin_ntp_servers=127.23.173.84:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:44.235203 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.224923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:41797 (local address 127.23.173.126:39253)
0927 22:19:44.225147 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.225153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.226834 (+ 1681us) server_negotiation.cc:407] Connection header received
0927 22:19:44.228670 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.228678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.228785 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.228907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:44.229911 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.230703 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.232239 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.232640 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.232775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.234303 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:44.234309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:44.234314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:44.234375 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:44.234414 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:44.234429 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:44.234539 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.234762 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.234861 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:19:44.236481 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.225585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39253 (local address 127.23.173.66:41797)
0927 22:19:44.226670 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:44.226714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:44.228413 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:44.229102 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:44.229109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:44.229194 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:44.229727 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.229743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.230862 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.230868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.232086 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.232097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.232857 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.232864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.233107 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.233912 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:44.233917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:44.233998 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:44.234089 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:44.234539 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:44.234552 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:44.234563 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:19:44.234753 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:19:44.235131 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":370,"threads_started":1}
I20260927 22:19:44.237897 9782 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39253
I20260927 22:19:44.238232 9782 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:44.239135 9782 heartbeater.cc:511] Master 127.23.173.126:39253 requested a full tablet report, sending...
I20260927 22:19:44.240625 9465 ts_manager.cc:194] Registered new tserver with Master: a1984138bf4e41b499ac008e2e32939b (127.23.173.66:35243)
I20260927 22:19:44.241422 9465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.66:41797
W20260927 22:19:44.559356 9785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:44.559633 9785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:44.559809 9785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:44.567121 9785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:44.567219 9785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:44.567389 9785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:44.576406 9785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:39253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:44.577683 9785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:44.579459 9785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:44.592121 9790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:44.592196 9791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:44.592233 9793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:44.593007 9785 server_base.cc:1034] running on GCE node
I20260927 22:19:44.593658 9785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:44.594740 9785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:44.595995 9785 hybrid_clock.cc:648] HybridClock initialized: now 1790547584595934 us; error 52 us; skew 500 ppm
I20260927 22:19:44.596412 9785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:44.598994 9785 webserver.cc:460] Webserver started at http://127.23.173.67:39665/ using document root <none> and password file <none>
I20260927 22:19:44.599720 9785 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:44.599864 9785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:44.600204 9785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:44.603359 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "456819f9b3854c71b1ace3c35d38e2d9"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "4db41a33f0a7c692f42cd439cb43526e"
server_key_iv: "6d52efec7fe29abd0eb7fa2f9c7d0120"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.604327 9785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "456819f9b3854c71b1ace3c35d38e2d9"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "4db41a33f0a7c692f42cd439cb43526e"
server_key_iv: "6d52efec7fe29abd0eb7fa2f9c7d0120"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.609503 9785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:19:44.612874 9799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.614244 9785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:44.614374 9785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "456819f9b3854c71b1ace3c35d38e2d9"
format_stamp: "Formatted at 2026-09-27 22:19:44 on dist-test-slave-3wc3"
server_key: "4db41a33f0a7c692f42cd439cb43526e"
server_key_iv: "6d52efec7fe29abd0eb7fa2f9c7d0120"
server_key_version: "encryptionkey@0"
I20260927 22:19:44.614547 9785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:44.627317 9785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:44.628180 9785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:44.628510 9785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:44.629489 9785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:44.630847 9785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:44.630931 9785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.631069 9785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:44.631171 9785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:44.665392 9785 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:44289
I20260927 22:19:44.665437 9911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:44289 every 8 connection(s)
I20260927 22:19:44.666719 9785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:44.675596 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9785
I20260927 22:19:44.675799 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:44.676425 24245 external_mini_cluster.cc:1428] Setting key 679e3019da8decb8de06fe13e1697844
I20260927 22:19:44.680365 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.669077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:42577 (local address 127.23.173.126:39253)
0927 22:19:44.669251 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.669256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.670603 (+ 1347us) server_negotiation.cc:407] Connection header received
0927 22:19:44.672496 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.672511 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.672611 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.672757 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:44.673889 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.674820 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.677270 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.677889 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.678088 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.679485 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:44.679516 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:44.679520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:44.679541 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:44.679578 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:44.679594 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:44.679681 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.679908 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.680003 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:19:44.681304 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.669554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:39253 (local address 127.23.173.67:42577)
0927 22:19:44.670448 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:44.670483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:44.672077 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:44.673010 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:44.673019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:44.673108 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:44.673694 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.673712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.674984 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.674990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.677110 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.677126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.678165 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.678172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.678404 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.679153 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:44.679156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:44.679207 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:44.679298 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:44.679681 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:44.679694 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:44.679704 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:19:44.679886 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 22:19:44.680247 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":260,"threads_started":1}
I20260927 22:19:44.682636 9912 heartbeater.cc:347] Connected to a master server at 127.23.173.126:39253
I20260927 22:19:44.682950 9912 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:44.683708 9912 heartbeater.cc:511] Master 127.23.173.126:39253 requested a full tablet report, sending...
I20260927 22:19:44.685098 9465 ts_manager.cc:194] Registered new tserver with Master: 456819f9b3854c71b1ace3c35d38e2d9 (127.23.173.67:44289)
I20260927 22:19:44.685885 9465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.67:42577
I20260927 22:19:44.691910 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:44.702312 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.696201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.23.173.126:39253)
0927 22:19:44.696383 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.696390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.696855 (+ 465us) server_negotiation.cc:407] Connection header received
0927 22:19:44.697027 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.697031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.697107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.697196 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:44.697732 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.698658 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.700017 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.700516 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.700617 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.701649 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:44.701654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:44.701657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:44.701675 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:44.701707 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:44.701720 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:44.701803 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.701923 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.702066 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:19:44.706414 9465 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:44.713744 9465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:44.755077 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.727244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:35243 (local address 127.0.0.1:55590)
0927 22:19:44.745370 (+ 18126us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:44.745406 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:44.745579 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:44.746653 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:44.746657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:44.746680 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:44.747218 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.747231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749303 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.749309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.751704 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.751727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.753897 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.753905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.754146 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.754200 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:19:44.754336 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:19:44.754545 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17948,"mutex_wait_us":113,"thread_start_us":17727,"threads_started":1}
I20260927 22:19:44.755390 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.727995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.23.173.66:35243)
0927 22:19:44.744768 (+ 16773us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.744783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.745746 (+ 963us) server_negotiation.cc:407] Connection header received
0927 22:19:44.745878 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.745886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.746199 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.746414 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:44.747418 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749155 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.751969 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.753665 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.753950 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.754379 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.754462 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.754670 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16589,"thread_start_us":16469,"threads_started":1}
I20260927 22:19:44.756255 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.727533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.23.173.65:46817)
0927 22:19:44.738546 (+ 11013us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.738570 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.744484 (+ 5914us) server_negotiation.cc:407] Connection header received
0927 22:19:44.745537 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.745546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.745849 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.746079 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:44.747259 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749423 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.752847 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.753877 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.754037 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.754312 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.755720 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.755906 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10747,"thread_start_us":10595,"threads_started":1}
I20260927 22:19:44.756968 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.726692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:44289 (local address 127.0.0.1:43846)
0927 22:19:44.743752 (+ 17060us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:44.743827 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:44.744088 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:44.746882 (+ 2794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:44.746892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:44.746918 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:44.747461 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.747477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749492 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.749501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.752569 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.752586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.754815 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.754824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.755078 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.755113 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:19:44.755207 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:19:44.755350 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16689,"mutex_wait_us":212,"thread_start_us":16447,"threads_started":1}
I20260927 22:19:44.757460 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.726974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.23.173.67:44289)
0927 22:19:44.746372 (+ 19398us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:44.746386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:44.746413 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:19:44.746490 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:44.746494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:44.746651 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:44.746889 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:44.747698 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749342 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.752824 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.754499 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.755911 (+ 1412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.756220 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:44.756297 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:19:44.756463 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19211,"thread_start_us":19041,"threads_started":1}
I20260927 22:19:44.759028 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:44.726868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:46817 (local address 127.0.0.1:40286)
0927 22:19:44.744302 (+ 17434us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:44.744329 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:44.745257 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:44.746386 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:44.746394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:44.746427 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:44.746910 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.746924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.749547 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.749555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.752569 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:44.752586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:44.754511 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:44.754518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:44.754684 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:44.754722 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:19:44.754821 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:19:44.754966 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17224,"mutex_wait_us":116,"thread_start_us":17654,"threads_started":1}
I20260927 22:19:44.759721 9717 tablet_service.cc:1467] Processing CreateTablet for tablet 31d4ab0cc367460d856cba3237167995 (DEFAULT_TABLE table=test-table [id=98732db91cd64dfb9652025240edffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:44.761293 9717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d4ab0cc367460d856cba3237167995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:44.761272 9847 tablet_service.cc:1467] Processing CreateTablet for tablet 31d4ab0cc367460d856cba3237167995 (DEFAULT_TABLE table=test-table [id=98732db91cd64dfb9652025240edffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:44.763135 9847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d4ab0cc367460d856cba3237167995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:44.763381 9586 tablet_service.cc:1467] Processing CreateTablet for tablet 31d4ab0cc367460d856cba3237167995 (DEFAULT_TABLE table=test-table [id=98732db91cd64dfb9652025240edffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:44.764828 9586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31d4ab0cc367460d856cba3237167995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:44.775197 9930 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Bootstrap starting.
I20260927 22:19:44.778578 9930 tablet_bootstrap.cc:654] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:44.779568 9931 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Bootstrap starting.
I20260927 22:19:44.781313 9930 log.cc:824] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:44.782761 9932 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Bootstrap starting.
I20260927 22:19:44.783782 9931 tablet_bootstrap.cc:654] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:44.784207 9651 heartbeater.cc:503] Master 127.23.173.126:39253 was elected leader, sending a full tablet report...
I20260927 22:19:44.785122 9931 log.cc:824] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:44.787096 9932 tablet_bootstrap.cc:654] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:44.787330 9930 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: No bootstrap required, opened a new log
I20260927 22:19:44.787642 9930 ts_tablet_manager.cc:1397] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260927 22:19:44.788555 9932 log.cc:824] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:44.794041 9932 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: No bootstrap required, opened a new log
I20260927 22:19:44.794178 9931 tablet_bootstrap.cc:492] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: No bootstrap required, opened a new log
I20260927 22:19:44.794407 9932 ts_tablet_manager.cc:1397] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 22:19:44.794479 9931 ts_tablet_manager.cc:1397] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260927 22:19:44.800019 9930 raft_consensus.cc:348] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.800503 9930 raft_consensus.cc:374] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:44.800618 9930 raft_consensus.cc:729] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1984138bf4e41b499ac008e2e32939b, State: Initialized, Role: FOLLOWER
I20260927 22:19:44.801383 9930 consensus_queue.cc:260] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.802783 9782 heartbeater.cc:503] Master 127.23.173.126:39253 was elected leader, sending a full tablet report...
I20260927 22:19:44.802927 9932 raft_consensus.cc:348] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.803368 9932 raft_consensus.cc:374] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:44.803522 9932 raft_consensus.cc:729] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aa4f889d55f47949689e93ce03e54ac, State: Initialized, Role: FOLLOWER
I20260927 22:19:44.803565 9930 ts_tablet_manager.cc:1428] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260927 22:19:44.804199 9932 consensus_queue.cc:260] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.805825 9931 raft_consensus.cc:348] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.806241 9932 ts_tablet_manager.cc:1428] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260927 22:19:44.806315 9931 raft_consensus.cc:374] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:44.806457 9931 raft_consensus.cc:729] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456819f9b3854c71b1ace3c35d38e2d9, State: Initialized, Role: FOLLOWER
I20260927 22:19:44.807076 9931 consensus_queue.cc:260] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:44.808182 9912 heartbeater.cc:503] Master 127.23.173.126:39253 was elected leader, sending a full tablet report...
I20260927 22:19:44.809351 9931 ts_tablet_manager.cc:1428] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260927 22:19:44.918386 9913 tablet.cc:2369] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:44.974269 9783 tablet.cc:2369] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:45.012450 9652 tablet.cc:2369] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:45.210321 9937 raft_consensus.cc:482] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:45.210662 9937 raft_consensus.cc:504] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:45.212424 9937 leader_election.cc:288] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 456819f9b3854c71b1ace3c35d38e2d9 (127.23.173.67:44289), a1984138bf4e41b499ac008e2e32939b (127.23.173.66:35243)
I20260927 22:19:45.219309 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.213072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:47655 (local address 127.23.173.66:35243)
0927 22:19:45.213226 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.213231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.213999 (+ 768us) server_negotiation.cc:407] Connection header received
0927 22:19:45.214183 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.214188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.214267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.214367 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:45.214994 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.215671 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.217517 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.218419 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.218587 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.218655 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.218917 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.219051 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 22:19:45.219631 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.212732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:53159 (local address 127.23.173.67:44289)
0927 22:19:45.212949 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.212956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.213888 (+ 932us) server_negotiation.cc:407] Connection header received
0927 22:19:45.214095 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.214101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.214170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.214269 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:45.214889 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.215593 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.217556 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.218509 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.218704 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.218794 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.219276 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.219411 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:19:45.219656 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.212944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:35243 (local address 127.23.173.65:47655)
0927 22:19:45.213870 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:45.213889 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:45.214043 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:45.214385 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:45.214389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:45.214411 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:45.214842 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:45.214852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.215812 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.215817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:45.217312 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:45.217328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.218598 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.218604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:45.218756 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.218778 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:19:45.218889 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:19:45.219064 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":49,"thread_start_us":587,"threads_started":1}
I20260927 22:19:45.219784 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.212764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:44289 (local address 127.23.173.65:53159)
0927 22:19:45.213751 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:45.213781 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:45.213907 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:45.214285 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:45.214289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:45.214360 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:45.214733 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:45.214743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.215730 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.215738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:45.217338 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:45.217349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.218914 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.218918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:45.219125 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.219148 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:19:45.219251 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:19:45.219363 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":94,"thread_start_us":487,"threads_started":1}
I20260927 22:19:45.220870 9867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d4ab0cc367460d856cba3237167995" candidate_uuid: "8aa4f889d55f47949689e93ce03e54ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456819f9b3854c71b1ace3c35d38e2d9" is_pre_election: true
I20260927 22:19:45.220865 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d4ab0cc367460d856cba3237167995" candidate_uuid: "8aa4f889d55f47949689e93ce03e54ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1984138bf4e41b499ac008e2e32939b" is_pre_election: true
I20260927 22:19:45.221392 9867 raft_consensus.cc:2454] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aa4f889d55f47949689e93ce03e54ac in term 0.
I20260927 22:19:45.221405 9737 raft_consensus.cc:2454] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aa4f889d55f47949689e93ce03e54ac in term 0.
I20260927 22:19:45.222105 9539 leader_election.cc:302] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456819f9b3854c71b1ace3c35d38e2d9, 8aa4f889d55f47949689e93ce03e54ac; no voters:
I20260927 22:19:45.222524 9937 raft_consensus.cc:2781] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:45.222674 9937 raft_consensus.cc:482] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:45.222781 9937 raft_consensus.cc:3037] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:45.224942 9937 raft_consensus.cc:504] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:45.225637 9937 leader_election.cc:288] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [CANDIDATE]: Term 1 election: Requested vote from peers 456819f9b3854c71b1ace3c35d38e2d9 (127.23.173.67:44289), a1984138bf4e41b499ac008e2e32939b (127.23.173.66:35243)
I20260927 22:19:45.226038 9867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d4ab0cc367460d856cba3237167995" candidate_uuid: "8aa4f889d55f47949689e93ce03e54ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456819f9b3854c71b1ace3c35d38e2d9"
I20260927 22:19:45.226150 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d4ab0cc367460d856cba3237167995" candidate_uuid: "8aa4f889d55f47949689e93ce03e54ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1984138bf4e41b499ac008e2e32939b"
I20260927 22:19:45.226249 9867 raft_consensus.cc:3037] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:45.226359 9737 raft_consensus.cc:3037] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:45.228402 9867 raft_consensus.cc:2454] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aa4f889d55f47949689e93ce03e54ac in term 1.
I20260927 22:19:45.228770 9539 leader_election.cc:302] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456819f9b3854c71b1ace3c35d38e2d9, 8aa4f889d55f47949689e93ce03e54ac; no voters:
I20260927 22:19:45.229048 9937 raft_consensus.cc:2781] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:45.229563 9937 raft_consensus.cc:686] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 1 LEADER]: Becoming Leader. State: Replica: 8aa4f889d55f47949689e93ce03e54ac, State: Running, Role: LEADER
I20260927 22:19:45.229612 9737 raft_consensus.cc:2454] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aa4f889d55f47949689e93ce03e54ac in term 1.
I20260927 22:19:45.230639 9937 consensus_queue.cc:237] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } }
I20260927 22:19:45.235285 9464 catalog_manager.cc:5495] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aa4f889d55f47949689e93ce03e54ac (127.23.173.65). New cstate: current_term: 1 leader_uuid: "8aa4f889d55f47949689e93ce03e54ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aa4f889d55f47949689e93ce03e54ac" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:45.327072 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.320799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.23.173.65:46817)
0927 22:19:45.321396 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.321402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.321524 (+ 122us) server_negotiation.cc:407] Connection header received
0927 22:19:45.321775 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.321780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.321869 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.322084 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:45.322535 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.323220 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.324824 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.325228 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.325379 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.325813 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:45.326479 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:45.326639 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.326719 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.326795 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260927 22:19:45.335208 9737 raft_consensus.cc:1261] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 1 FOLLOWER]: Refusing update from remote peer 8aa4f889d55f47949689e93ce03e54ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:45.335292 9867 raft_consensus.cc:1261] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 1 FOLLOWER]: Refusing update from remote peer 8aa4f889d55f47949689e93ce03e54ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:45.336241 9941 consensus_queue.cc:1035] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1984138bf4e41b499ac008e2e32939b" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:45.336565 9937 consensus_queue.cc:1035] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "456819f9b3854c71b1ace3c35d38e2d9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:45.349996 9953 mvcc.cc:205] Tried to move back new op lower bound from 7334082909521727488 to 7334082909108715520. Current Snapshot: MvccSnapshot[applied={T|T < 7334082909521727488}]
I20260927 22:19:45.353214 9949 mvcc.cc:205] Tried to move back new op lower bound from 7334082909521727488 to 7334082909108715520. Current Snapshot: MvccSnapshot[applied={T|T < 7334082909521727488 or (T in {7334082909521727488})}]
I20260927 22:19:45.362377 9465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:45.362612 9465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:45.366106 9465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c97c0ebae5524dc48eb1ceeac1683f5b: Sending DeleteTablet for 3 replicas of tablet 31d4ab0cc367460d856cba3237167995
I20260927 22:19:45.367218 9586 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d4ab0cc367460d856cba3237167995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:45 UTC) from {username='slave'} at 127.0.0.1:40286
I20260927 22:19:45.367221 9847 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d4ab0cc367460d856cba3237167995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:45 UTC) from {username='slave'} at 127.0.0.1:43846
I20260927 22:19:45.367410 9717 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d4ab0cc367460d856cba3237167995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:45 UTC) from {username='slave'} at 127.0.0.1:55590
I20260927 22:19:45.368477 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9524
I20260927 22:19:45.381148 9962 tablet_replica.cc:330] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: stopping tablet replica
I20260927 22:19:45.381732 9962 raft_consensus.cc:2229] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:45.382586 9962 raft_consensus.cc:2258] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:45.385159 9962 ts_tablet_manager.cc:1905] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:45.394961 9962 ts_tablet_manager.cc:1918] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:45.395157 9962 log.cc:1196] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/31d4ab0cc367460d856cba3237167995
I20260927 22:19:45.395573 9961 tablet_replica.cc:330] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: stopping tablet replica
I20260927 22:19:45.396127 9962 ts_tablet_manager.cc:1939] T 31d4ab0cc367460d856cba3237167995 P 8aa4f889d55f47949689e93ce03e54ac: Deleting consensus metadata
I20260927 22:19:45.396159 9961 raft_consensus.cc:2229] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:45.396773 9961 raft_consensus.cc:2258] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:45.398290 9450 catalog_manager.cc:4841] TS 8aa4f889d55f47949689e93ce03e54ac (127.23.173.65:46817): tablet 31d4ab0cc367460d856cba3237167995 (table test-table [id=98732db91cd64dfb9652025240edffc7]) successfully deleted
I20260927 22:19:45.399276 9961 ts_tablet_manager.cc:1905] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:45.400480 9963 tablet_replica.cc:330] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: stopping tablet replica
I20260927 22:19:45.400926 9963 raft_consensus.cc:2229] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:45.401438 9963 raft_consensus.cc:2258] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:45.404062 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.369620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.23.173.65:46817)
0927 22:19:45.369826 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.369835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.393233 (+ 23398us) server_negotiation.cc:407] Connection header received
0927 22:19:45.393777 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.393784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.393899 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.394012 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:45.394764 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.396546 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.399496 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.400229 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.400334 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.402207 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:45.402219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:45.402237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:45.403016 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:45.403085 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:45.403112 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:45.403202 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.403505 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.403624 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:19:45.405274 9646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:45.406032 9963 ts_tablet_manager.cc:1905] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:45.408368 9961 ts_tablet_manager.cc:1918] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:45.408538 9961 log.cc:1196] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/31d4ab0cc367460d856cba3237167995
I20260927 22:19:45.409000 9961 ts_tablet_manager.cc:1939] T 31d4ab0cc367460d856cba3237167995 P 456819f9b3854c71b1ace3c35d38e2d9: Deleting consensus metadata
I20260927 22:19:45.410586 9449 catalog_manager.cc:4841] TS 456819f9b3854c71b1ace3c35d38e2d9 (127.23.173.67:44289): tablet 31d4ab0cc367460d856cba3237167995 (table test-table [id=98732db91cd64dfb9652025240edffc7]) successfully deleted
I20260927 22:19:45.414359 9963 ts_tablet_manager.cc:1918] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:45.414529 9963 log.cc:1196] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/31d4ab0cc367460d856cba3237167995
I20260927 22:19:45.415019 9963 ts_tablet_manager.cc:1939] T 31d4ab0cc367460d856cba3237167995 P a1984138bf4e41b499ac008e2e32939b: Deleting consensus metadata
I20260927 22:19:45.417163 9451 catalog_manager.cc:4841] TS a1984138bf4e41b499ac008e2e32939b (127.23.173.66:35243): tablet 31d4ab0cc367460d856cba3237167995 (table test-table [id=98732db91cd64dfb9652025240edffc7]) successfully deleted
I20260927 22:19:45.505651 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9524
I20260927 22:19:45.524277 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9655
I20260927 22:19:45.531527 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.525106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.23.173.66:35243)
0927 22:19:45.525331 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.525347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.525368 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:19:45.525572 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.525577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.525671 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.525793 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:45.526254 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.527172 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.528420 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.528832 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.528993 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.530134 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:45.530144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:45.530157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:45.530750 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:45.530796 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:45.530815 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:45.530952 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.531088 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.531192 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:19:45.532258 9777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:45.620131 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9655
I20260927 22:19:45.638566 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9785
I20260927 22:19:45.645668 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:45.639323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.23.173.67:44289)
0927 22:19:45.639498 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:45.639503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:45.639627 (+ 124us) server_negotiation.cc:407] Connection header received
0927 22:19:45.639801 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:45.639805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:45.639908 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:45.639993 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:45.640424 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.641138 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.642728 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:45.643194 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:45.643300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:45.644452 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:45.644460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:45.644475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:45.645030 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:45.645074 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:45.645094 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:45.645176 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:45.645319 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:19:45.645418 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:19:45.646407 9907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:45.735150 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9785
I20260927 22:19:45.751555 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9434
I20260927 22:19:45.752669 9495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:45.860910 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9434
2026-09-27T22:19:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:19:45 dist-test-slave-3wc3 krb5kdc[9971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:19:45 dist-test-slave-3wc3 krb5kdc[9971](info): set up 2 sockets
Sep 27 22:19:45 dist-test-slave-3wc3 krb5kdc[9971](info): commencing operation
krb5kdc: starting...
W20260927 22:19:46.990178 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:19:47 dist-test-slave-3wc3 krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:19:47.238270 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38493
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:47.563477 9984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:47.563807 9984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:47.563974 9984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:47.571210 9984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:47.571355 9984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:47.571463 9984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:47.571566 9984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:47.571645 9984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:47.581200 9984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:47.582533 9984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:47.583909 9984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:47.595741 9992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:47.595825 9990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:47.595939 9989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:47.596920 9984 server_base.cc:1034] running on GCE node
I20260927 22:19:47.597697 9984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:47.599140 9984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:47.600342 9984 hybrid_clock.cc:648] HybridClock initialized: now 1790547587600281 us; error 55 us; skew 500 ppm
I20260927 22:19:47.600696 9984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:47 dist-test-slave-3wc3 krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:47.603796 9984 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:19:47.605139 9984 webserver.cc:460] Webserver started at http://127.23.173.126:43009/ using document root <none> and password file <none>
I20260927 22:19:47.605777 9984 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:47.605875 9984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:47.606189 9984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:47.609077 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "07f455223f5b4e4cac8397b1e37aff90"
format_stamp: "Formatted at 2026-09-27 22:19:47 on dist-test-slave-3wc3"
server_key: "f392294f0364b55102bf977052b84e3d"
server_key_iv: "225cb3646f76fa8cd91c937276c427e7"
server_key_version: "encryptionkey@0"
I20260927 22:19:47.609741 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "07f455223f5b4e4cac8397b1e37aff90"
format_stamp: "Formatted at 2026-09-27 22:19:47 on dist-test-slave-3wc3"
server_key: "f392294f0364b55102bf977052b84e3d"
server_key_iv: "225cb3646f76fa8cd91c937276c427e7"
server_key_version: "encryptionkey@0"
I20260927 22:19:47.614797 9984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:19:47.618302 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:47.619604 9984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:19:47.619801 9984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "07f455223f5b4e4cac8397b1e37aff90"
format_stamp: "Formatted at 2026-09-27 22:19:47 on dist-test-slave-3wc3"
server_key: "f392294f0364b55102bf977052b84e3d"
server_key_iv: "225cb3646f76fa8cd91c937276c427e7"
server_key_version: "encryptionkey@0"
I20260927 22:19:47.620000 9984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:47.640484 9984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:47.644812 9984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:47.645118 9984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:47.663251 9984 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:38493
I20260927 22:19:47.663270 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:38493 every 8 connection(s)
I20260927 22:19:47.664623 9984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:47.668331 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:47.671173 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 9984
I20260927 22:19:47.671383 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:47.671828 24245 external_mini_cluster.cc:1428] Setting key d9b80365294e9f7b2895bd5a78926417
I20260927 22:19:47.681095 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: Bootstrap starting.
I20260927 22:19:47.684547 10051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:47.685526 10051 log.cc:824] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:19:47 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:47.688799 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: No bootstrap required, opened a new log
I20260927 22:19:47.696847 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:47.673759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.23.173.126:38493)
0927 22:19:47.674521 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:47.674535 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:47.674580 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:19:47.676882 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:47.676910 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:47.677439 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:47.677885 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:47.679315 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:47.681266 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:47.682620 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:47.683666 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:47.683854 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:47.688578 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:47.688586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:47.688601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:47.688635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:47.692427 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:47.693354 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:47.693363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:47.693374 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:47.693555 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:47.694054 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:47.694058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:47.694060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:47.694631 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:47.694858 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:47.695335 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 22:19:47.695751 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":344,"threads_started":1}
I20260927 22:19:47.698082 10051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } }
I20260927 22:19:47.698551 10051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:47.698669 10051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f455223f5b4e4cac8397b1e37aff90, State: Initialized, Role: FOLLOWER
I20260927 22:19:47.699306 10051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } }
I20260927 22:19:47.699569 10051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:47.699726 10051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:47.699913 10051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:47.702122 10051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } }
I20260927 22:19:47.702560 10051 leader_election.cc:302] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07f455223f5b4e4cac8397b1e37aff90; no voters:
I20260927 22:19:47.703276 10051 leader_election.cc:288] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:47.703583 10056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:47.704432 10056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [term 1 LEADER]: Becoming Leader. State: Replica: 07f455223f5b4e4cac8397b1e37aff90, State: Running, Role: LEADER
I20260927 22:19:47.704953 10056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } }
I20260927 22:19:47.705502 10051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:47.707872 10058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f455223f5b4e4cac8397b1e37aff90. Latest consensus state: current_term: 1 leader_uuid: "07f455223f5b4e4cac8397b1e37aff90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } } }
I20260927 22:19:47.708343 10058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:47.709111 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07f455223f5b4e4cac8397b1e37aff90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f455223f5b4e4cac8397b1e37aff90" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 38493 } } }
I20260927 22:19:47.709441 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:47.710450 10061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:47.714948 10061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:47.724606 10061 catalog_manager.cc:1347] Generated new cluster ID: 33ba82db26b2434eb319ce89c88673ed
I20260927 22:19:47.724781 10061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:47.738087 10061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:47.739944 10061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:47.751199 10061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: Generated new TSK 0
I20260927 22:19:47.752193 10061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:19:47.817838 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:48.154328 10077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:48.154641 10077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:48.154824 10077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:48.161924 10077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:48.162109 10077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:48.162298 10077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:48.172016 10077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:48.173316 10077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:48.174933 10077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:48.187788 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:48.187882 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:48.188071 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:48.188603 10077 server_base.cc:1034] running on GCE node
I20260927 22:19:48.189195 10077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:48.190171 10077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:48.191404 10077 hybrid_clock.cc:648] HybridClock initialized: now 1790547588191370 us; error 42 us; skew 500 ppm
I20260927 22:19:48.191718 10077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:48 dist-test-slave-3wc3 krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:48.202759 10077 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:19:48.204450 10077 webserver.cc:460] Webserver started at http://127.23.173.65:38407/ using document root <none> and password file <none>
I20260927 22:19:48.205226 10077 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:48.205382 10077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:48.205717 10077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:48.209007 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "4357a3ae5bb6404b82d7c8133bc50cea"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "438dbfef388578b0172117706f64b1c1"
server_key_iv: "662e98fb3eff6633305af4e82dec8e9f"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.209830 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "4357a3ae5bb6404b82d7c8133bc50cea"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "438dbfef388578b0172117706f64b1c1"
server_key_iv: "662e98fb3eff6633305af4e82dec8e9f"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.215164 10077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:19:48.218696 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.219898 10077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:48.220065 10077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "4357a3ae5bb6404b82d7c8133bc50cea"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "438dbfef388578b0172117706f64b1c1"
server_key_iv: "662e98fb3eff6633305af4e82dec8e9f"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.220270 10077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:48.234408 10077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:48.238656 10077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:48.238967 10077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:48.239981 10077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:48.241281 10077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:48.241394 10077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.241549 10077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:48.241623 10077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.275729 10077 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:45735
I20260927 22:19:48.275763 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:45735 every 8 connection(s)
I20260927 22:19:48.277100 10077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:48.284806 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10077
I20260927 22:19:48.284994 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:48.285426 24245 external_mini_cluster.cc:1428] Setting key 69a795c512af529a3d0b3d5a454e9beb
Sep 27 22:19:48 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:48.299775 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:48.279724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:57035 (local address 127.23.173.126:38493)
0927 22:19:48.280188 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:48.280193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:48.281101 (+ 908us) server_negotiation.cc:407] Connection header received
0927 22:19:48.282995 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:48.283001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:48.283098 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:48.283218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:48.285269 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.286100 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.287858 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.288562 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.288713 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:48.292472 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:48.292481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:48.292489 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:48.292536 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:48.296112 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:48.297178 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:48.297182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:48.297185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:48.297306 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:48.297897 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:48.297901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:48.297903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:48.298254 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:48.298377 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:48.299333 (+ 956us) server_negotiation.cc:299] Negotiation successful
0927 22:19:48.299517 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260927 22:19:48.301419 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:48.280259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38493 (local address 127.23.173.65:57035)
0927 22:19:48.280946 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:48.280987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:48.282726 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:48.283535 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:48.283547 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:48.284393 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:48.285099 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:48.285123 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.286289 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.286293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:48.287713 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:48.287724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.288769 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.288774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:48.288971 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:48.289701 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:48.289725 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:48.292239 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:48.296264 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:48.296277 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:48.296296 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:48.297034 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:48.297480 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:48.297489 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:48.297496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:48.297751 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:48.298391 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:48.298403 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:48.298912 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 22:19:48.299235 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 22:19:48.299861 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":259,"threads_started":1}
I20260927 22:19:48.303112 10205 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38493
I20260927 22:19:48.303449 10205 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:48.304168 10205 heartbeater.cc:511] Master 127.23.173.126:38493 requested a full tablet report, sending...
I20260927 22:19:48.306499 10016 ts_manager.cc:194] Registered new tserver with Master: 4357a3ae5bb6404b82d7c8133bc50cea (127.23.173.65:45735)
I20260927 22:19:48.308032 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:57035
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:19:48.346369 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:48.668476 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:48.668781 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:48.668953 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:48.676340 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:48.676496 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:48.676693 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:48.686365 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:48.687579 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:48.688912 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:48.702106 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:48.702142 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:48.702221 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:48.703015 10211 server_base.cc:1034] running on GCE node
I20260927 22:19:48.703603 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:48.704596 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:48.705816 10211 hybrid_clock.cc:648] HybridClock initialized: now 1790547588705753 us; error 59 us; skew 500 ppm
I20260927 22:19:48.706182 10211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:48 dist-test-slave-3wc3 krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:48.709306 10211 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:19:48.710747 10211 webserver.cc:460] Webserver started at http://127.23.173.66:36089/ using document root <none> and password file <none>
I20260927 22:19:48.711372 10211 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:48.711479 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:48.711730 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:48.714563 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "0f727ef7d79b4f32af897c7b93083ec5"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "bf4487654c8f08e2e2f848280756ec68"
server_key_iv: "8da7e24746073c7114afba57c8ee57e8"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.715240 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "0f727ef7d79b4f32af897c7b93083ec5"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "bf4487654c8f08e2e2f848280756ec68"
server_key_iv: "8da7e24746073c7114afba57c8ee57e8"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.720292 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:19:48.723758 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.725198 10211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:19:48.725350 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "0f727ef7d79b4f32af897c7b93083ec5"
format_stamp: "Formatted at 2026-09-27 22:19:48 on dist-test-slave-3wc3"
server_key: "bf4487654c8f08e2e2f848280756ec68"
server_key_iv: "8da7e24746073c7114afba57c8ee57e8"
server_key_version: "encryptionkey@0"
I20260927 22:19:48.725587 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:48.752756 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:48.756726 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:48.757015 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:48.758029 10211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:48.759342 10211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:48.759414 10211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.759590 10211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:48.759666 10211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:48.792675 10211 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:39513
I20260927 22:19:48.792761 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:39513 every 8 connection(s)
I20260927 22:19:48.794035 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:48.800556 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10211
I20260927 22:19:48.800737 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:48.801174 24245 external_mini_cluster.cc:1428] Setting key 956ead4f66a522c8c8d262022d7cc642
Sep 27 22:19:48 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:48.813882 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:48.796569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36255 (local address 127.23.173.126:38493)
0927 22:19:48.796758 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:48.796762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:48.797873 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 22:19:48.799735 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:48.799740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:48.799833 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:48.799947 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:48.801587 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.802578 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.804095 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.804531 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.804673 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:48.808304 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:48.808309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:48.808312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:48.808343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:48.810879 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:48.811595 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:48.811602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:48.811605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:48.811734 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:48.812159 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:48.812166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:48.812172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:48.812425 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:48.812589 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:48.813271 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:19:48.813543 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:19:48.814867 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:48.797040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38493 (local address 127.23.173.66:36255)
0927 22:19:48.797707 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:48.797743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:48.799462 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:48.800184 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:48.800192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:48.800751 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:48.801422 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:48.801443 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.802720 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.802728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:48.803933 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:48.803962 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:48.804756 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:48.804763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:48.805008 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:48.805780 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:48.805803 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:48.808083 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:48.811024 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:48.811035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:48.811052 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:48.811473 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:48.811845 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:48.811850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:48.811854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:48.812044 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:48.812558 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:48.812567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:48.812908 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 22:19:48.813187 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 22:19:48.813718 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":261,"threads_started":1}
I20260927 22:19:48.816346 10339 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38493
I20260927 22:19:48.816673 10339 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:48.817535 10339 heartbeater.cc:511] Master 127.23.173.126:38493 requested a full tablet report, sending...
I20260927 22:19:48.819227 10016 ts_manager.cc:194] Registered new tserver with Master: 0f727ef7d79b4f32af897c7b93083ec5 (127.23.173.66:39513)
I20260927 22:19:48.820492 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:36255
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:19:48.857664 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:49.183482 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:49.183745 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:49.183917 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:49.191160 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:49.191293 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:49.191470 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:49.200805 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:38493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:49.202142 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:49.203439 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:49.216260 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:49.216235 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:49.216449 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:49.217101 10344 server_base.cc:1034] running on GCE node
I20260927 22:19:49.217617 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:49.218513 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:49.219707 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790547589219657 us; error 37 us; skew 500 ppm
I20260927 22:19:49.220033 10344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:49 dist-test-slave-3wc3 krb5kdc[9971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:49.223284 10344 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:19:49.224910 10344 webserver.cc:460] Webserver started at http://127.23.173.67:39143/ using document root <none> and password file <none>
I20260927 22:19:49.225555 10344 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:49.225666 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:49.225920 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:49.228696 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "ee7c413b6ff648e09a2f4ff371d588c9"
format_stamp: "Formatted at 2026-09-27 22:19:49 on dist-test-slave-3wc3"
server_key: "64d3ba6701893f5911dbaf4f6a4ef512"
server_key_iv: "fc088c7b211d7541006570cf29b7379b"
server_key_version: "encryptionkey@0"
I20260927 22:19:49.229485 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "ee7c413b6ff648e09a2f4ff371d588c9"
format_stamp: "Formatted at 2026-09-27 22:19:49 on dist-test-slave-3wc3"
server_key: "64d3ba6701893f5911dbaf4f6a4ef512"
server_key_iv: "fc088c7b211d7541006570cf29b7379b"
server_key_version: "encryptionkey@0"
I20260927 22:19:49.234544 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:19:49.237783 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:49.239085 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:49.239248 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "ee7c413b6ff648e09a2f4ff371d588c9"
format_stamp: "Formatted at 2026-09-27 22:19:49 on dist-test-slave-3wc3"
server_key: "64d3ba6701893f5911dbaf4f6a4ef512"
server_key_iv: "fc088c7b211d7541006570cf29b7379b"
server_key_version: "encryptionkey@0"
I20260927 22:19:49.239445 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:49.282310 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:49.286343 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:49.286650 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:49.287622 10344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:49.288949 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:49.289021 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:49.289201 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:49.289253 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:49.311160 10205 heartbeater.cc:503] Master 127.23.173.126:38493 was elected leader, sending a full tablet report...
I20260927 22:19:49.322942 10344 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:40469
I20260927 22:19:49.322988 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:40469 every 8 connection(s)
I20260927 22:19:49.324404 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:49.333711 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10344
I20260927 22:19:49.333890 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:49.334457 24245 external_mini_cluster.cc:1428] Setting key 4ef9904d2ba315733bf185654064df38
Sep 27 22:19:49 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:49.343920 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.326868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:58713 (local address 127.23.173.126:38493)
0927 22:19:49.327333 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.327337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.328058 (+ 721us) server_negotiation.cc:407] Connection header received
0927 22:19:49.330107 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.330112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.330184 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.330274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:49.331659 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.332468 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.334302 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.334751 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.334879 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.338612 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:49.338616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:49.338619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:49.338650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:49.341248 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.341868 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.341872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.341874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.342055 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.342477 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.342480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.342482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.342738 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:49.342845 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.343497 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.343665 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260927 22:19:49.344846 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.327267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:38493 (local address 127.23.173.67:58713)
0927 22:19:49.327900 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.327949 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.329759 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.330486 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.330494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.330979 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:49.331505 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.331521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.332598 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.332602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.334164 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.334180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.334957 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.334962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.335221 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.336008 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:49.336038 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:49.338416 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:49.341357 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:49.341363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:49.341373 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:49.341763 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:49.342162 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:49.342165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:49.342168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:49.342327 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:49.342842 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:49.342847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:49.343120 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.343343 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.343806 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":248,"threads_started":1}
I20260927 22:19:49.346385 10472 heartbeater.cc:347] Connected to a master server at 127.23.173.126:38493
I20260927 22:19:49.346681 10472 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:49.347370 10472 heartbeater.cc:511] Master 127.23.173.126:38493 requested a full tablet report, sending...
I20260927 22:19:49.348958 10016 ts_manager.cc:194] Registered new tserver with Master: ee7c413b6ff648e09a2f4ff371d588c9 (127.23.173.67:40469)
I20260927 22:19:49.349745 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:49.350020 10016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:58713
I20260927 22:19:49.366963 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.354071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.23.173.126:38493)
0927 22:19:49.354295 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.354301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.354624 (+ 323us) server_negotiation.cc:407] Connection header received
0927 22:19:49.354769 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.354772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.354828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.354908 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:49.356463 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.357166 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.358689 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.359105 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.359195 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.361892 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:49.361897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:49.361900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:49.361930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:49.364364 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.365079 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.365083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.365085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.365208 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.365591 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.365594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.365596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.365867 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:49.366122 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.366453 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.366592 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:19:49.370903 10016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:49.378443 10016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:49.404707 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.391508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:45735 (local address 127.0.0.1:37124)
0927 22:19:49.393370 (+ 1862us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.393391 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.393527 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.394969 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.394981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.395024 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:49.395567 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.395584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.399180 (+ 3596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.399188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.401686 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.401705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.403709 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.403717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.403917 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.403984 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.404092 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.404273 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":1603,"threads_started":1}
I20260927 22:19:49.406224 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.392476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.23.173.65:45735)
0927 22:19:49.394150 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.394163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.394201 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:19:49.394324 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.394335 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.394660 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.395808 (+ 1148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:49.397330 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.399018 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.401898 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.403481 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.405286 (+ 1805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.405570 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.405646 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.405820 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":1305,"threads_started":1}
I20260927 22:19:49.408077 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.392007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.23.173.67:40469)
0927 22:19:49.397317 (+ 5310us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.397325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.397358 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:19:49.397451 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.397458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.397705 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.397893 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:49.398809 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.401626 (+ 2817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.405256 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.406859 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.407043 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.407269 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.407582 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.407740 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5157,"thread_start_us":5015,"threads_started":1}
I20260927 22:19:49.408070 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.391428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:40469 (local address 127.0.0.1:50142)
0927 22:19:49.392288 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.392341 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.392550 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.398198 (+ 5648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.398203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.398227 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:49.398596 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.398607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.401790 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.401798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.405083 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.405093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.407097 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.407103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.407273 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.407397 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.407541 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.407676 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":67,"thread_start_us":359,"threads_started":1}
I20260927 22:19:49.410862 10140 tablet_service.cc:1467] Processing CreateTablet for tablet 90479c8cd05145e1ad30a630046a71d4 (DEFAULT_TABLE table=test-table [id=5b4f9746b83c442ea2c5a33eb8491741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:49.411578 10407 tablet_service.cc:1467] Processing CreateTablet for tablet 90479c8cd05145e1ad30a630046a71d4 (DEFAULT_TABLE table=test-table [id=5b4f9746b83c442ea2c5a33eb8491741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:49.412585 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90479c8cd05145e1ad30a630046a71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:49.412827 10407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90479c8cd05145e1ad30a630046a71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:49.414076 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.393914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:39513 (local address 127.0.0.1:35160)
0927 22:19:49.404945 (+ 11031us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.404965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.405082 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.407220 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.407225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.407251 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:49.407517 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.407528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.409629 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.409634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.411445 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.411459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.413260 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.413266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.413442 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.413476 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.413576 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.413720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10934,"mutex_wait_us":2,"thread_start_us":12708,"threads_started":1}
I20260927 22:19:49.414175 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.394185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.23.173.66:39513)
0927 22:19:49.406540 (+ 12355us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.406553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.406590 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:19:49.406696 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.406703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.406977 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.407707 (+ 730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:49.407974 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.409454 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.411656 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.413086 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.413231 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.413401 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.413637 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.413805 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12152,"thread_start_us":12576,"threads_started":1}
I20260927 22:19:49.417834 10274 tablet_service.cc:1467] Processing CreateTablet for tablet 90479c8cd05145e1ad30a630046a71d4 (DEFAULT_TABLE table=test-table [id=5b4f9746b83c442ea2c5a33eb8491741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:49.419271 10274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90479c8cd05145e1ad30a630046a71d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:49.427150 10492 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Bootstrap starting.
I20260927 22:19:49.430701 10491 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Bootstrap starting.
I20260927 22:19:49.431995 10492 tablet_bootstrap.cc:654] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:49.433091 10493 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Bootstrap starting.
I20260927 22:19:49.433548 10492 log.cc:824] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:49.434168 10491 tablet_bootstrap.cc:654] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:49.435375 10491 log.cc:824] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:49.436317 10493 tablet_bootstrap.cc:654] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:49.436930 10492 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: No bootstrap required, opened a new log
I20260927 22:19:49.437215 10492 ts_tablet_manager.cc:1397] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:19:49.437485 10493 log.cc:824] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:49.438063 10491 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: No bootstrap required, opened a new log
I20260927 22:19:49.438331 10491 ts_tablet_manager.cc:1397] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 22:19:49.440572 10493 tablet_bootstrap.cc:492] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: No bootstrap required, opened a new log
I20260927 22:19:49.440812 10493 ts_tablet_manager.cc:1397] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:19:49.446877 10491 raft_consensus.cc:348] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.447270 10491 raft_consensus.cc:374] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:49.447363 10491 raft_consensus.cc:729] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7c413b6ff648e09a2f4ff371d588c9, State: Initialized, Role: FOLLOWER
I20260927 22:19:49.447996 10491 consensus_queue.cc:260] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.447990 10492 raft_consensus.cc:348] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.448423 10492 raft_consensus.cc:374] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:49.448524 10492 raft_consensus.cc:729] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4357a3ae5bb6404b82d7c8133bc50cea, State: Initialized, Role: FOLLOWER
I20260927 22:19:49.449191 10472 heartbeater.cc:503] Master 127.23.173.126:38493 was elected leader, sending a full tablet report...
I20260927 22:19:49.449177 10492 consensus_queue.cc:260] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.450954 10492 ts_tablet_manager.cc:1428] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 22:19:49.451408 10493 raft_consensus.cc:348] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.451756 10493 raft_consensus.cc:374] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:49.451853 10493 raft_consensus.cc:729] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f727ef7d79b4f32af897c7b93083ec5, State: Initialized, Role: FOLLOWER
I20260927 22:19:49.452405 10493 consensus_queue.cc:260] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.452688 10491 ts_tablet_manager.cc:1428] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260927 22:19:49.454619 10339 heartbeater.cc:503] Master 127.23.173.126:38493 was elected leader, sending a full tablet report...
I20260927 22:19:49.455258 10493 ts_tablet_manager.cc:1428] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 22:19:49.470281 10498 raft_consensus.cc:482] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:49.470536 10498 raft_consensus.cc:504] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.472193 10498 leader_election.cc:288] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee7c413b6ff648e09a2f4ff371d588c9 (127.23.173.67:40469), 0f727ef7d79b4f32af897c7b93083ec5 (127.23.173.66:39513)
I20260927 22:19:49.480786 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.472763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:45591 (local address 127.23.173.66:39513)
0927 22:19:49.472976 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.472982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.473566 (+ 584us) server_negotiation.cc:407] Connection header received
0927 22:19:49.473693 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.473701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.473767 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.473854 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:49.474573 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.475589 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.478488 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.479707 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.479857 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.479927 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.480412 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.480540 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:19:49.480794 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.472651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:44673 (local address 127.23.173.67:40469)
0927 22:19:49.473043 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.473052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.473305 (+ 253us) server_negotiation.cc:407] Connection header received
0927 22:19:49.473520 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.473547 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.473627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.473722 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:49.474383 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.475499 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.478488 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.479710 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.479861 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.479927 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.480412 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.480550 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260927 22:19:49.481050 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.472637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:39513 (local address 127.23.173.65:45591)
0927 22:19:49.473475 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.473489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.473586 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.473892 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.473896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.473918 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:49.474400 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.474411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.475712 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.475716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.478317 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.478337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.479964 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.479971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.480214 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.480277 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.480373 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.480580 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":289,"thread_start_us":520,"threads_started":1}
I20260927 22:19:49.481325 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.472379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:40469 (local address 127.23.173.65:44673)
0927 22:19:49.473178 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:49.473197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:49.473380 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:49.473730 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:49.473733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:49.473749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:49.474224 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.474239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.475635 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.475658 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.478334 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:49.478343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.479977 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.479983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:49.480214 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.480278 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:19:49.480373 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:19:49.480580 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":251,"thread_start_us":609,"threads_started":1}
I20260927 22:19:49.482060 10294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90479c8cd05145e1ad30a630046a71d4" candidate_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f727ef7d79b4f32af897c7b93083ec5" is_pre_election: true
I20260927 22:19:49.482306 10427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90479c8cd05145e1ad30a630046a71d4" candidate_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" is_pre_election: true
I20260927 22:19:49.482503 10294 raft_consensus.cc:2454] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4357a3ae5bb6404b82d7c8133bc50cea in term 0.
I20260927 22:19:49.482784 10427 raft_consensus.cc:2454] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4357a3ae5bb6404b82d7c8133bc50cea in term 0.
I20260927 22:19:49.483134 10096 leader_election.cc:302] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f727ef7d79b4f32af897c7b93083ec5, 4357a3ae5bb6404b82d7c8133bc50cea; no voters:
I20260927 22:19:49.483505 10498 raft_consensus.cc:2781] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:49.483649 10498 raft_consensus.cc:482] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:49.483747 10498 raft_consensus.cc:3037] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:49.485855 10498 raft_consensus.cc:504] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.486482 10498 leader_election.cc:288] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [CANDIDATE]: Term 1 election: Requested vote from peers ee7c413b6ff648e09a2f4ff371d588c9 (127.23.173.67:40469), 0f727ef7d79b4f32af897c7b93083ec5 (127.23.173.66:39513)
I20260927 22:19:49.486897 10427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90479c8cd05145e1ad30a630046a71d4" candidate_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee7c413b6ff648e09a2f4ff371d588c9"
I20260927 22:19:49.486965 10294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90479c8cd05145e1ad30a630046a71d4" candidate_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f727ef7d79b4f32af897c7b93083ec5"
I20260927 22:19:49.487099 10427 raft_consensus.cc:3037] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:49.487133 10294 raft_consensus.cc:3037] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:49.490286 10294 raft_consensus.cc:2454] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4357a3ae5bb6404b82d7c8133bc50cea in term 1.
I20260927 22:19:49.490293 10427 raft_consensus.cc:2454] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4357a3ae5bb6404b82d7c8133bc50cea in term 1.
I20260927 22:19:49.490706 10096 leader_election.cc:302] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f727ef7d79b4f32af897c7b93083ec5, 4357a3ae5bb6404b82d7c8133bc50cea, ee7c413b6ff648e09a2f4ff371d588c9; no voters:
I20260927 22:19:49.490988 10498 raft_consensus.cc:2781] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:49.491439 10498 raft_consensus.cc:686] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 1 LEADER]: Becoming Leader. State: Replica: 4357a3ae5bb6404b82d7c8133bc50cea, State: Running, Role: LEADER
I20260927 22:19:49.491941 10498 consensus_queue.cc:237] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } }
I20260927 22:19:49.496479 10014 catalog_manager.cc:5495] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4357a3ae5bb6404b82d7c8133bc50cea (127.23.173.65). New cstate: current_term: 1 leader_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4357a3ae5bb6404b82d7c8133bc50cea" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 45735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:49.528417 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.523088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.23.173.65:45735)
0927 22:19:49.523278 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.523284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.523334 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:19:49.523528 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.523532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.523622 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.523744 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:49.524142 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.524938 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.526403 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.526786 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.526884 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.527321 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:49.527788 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:49.527926 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.528051 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.528124 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 22:19:49.529572 10206 tablet.cc:2369] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:49.536544 10427 raft_consensus.cc:1261] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 1 FOLLOWER]: Refusing update from remote peer 4357a3ae5bb6404b82d7c8133bc50cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:49.536545 10294 raft_consensus.cc:1261] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 1 FOLLOWER]: Refusing update from remote peer 4357a3ae5bb6404b82d7c8133bc50cea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:49.537525 10502 consensus_queue.cc:1035] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f727ef7d79b4f32af897c7b93083ec5" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 39513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:49.537873 10498 consensus_queue.cc:1035] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee7c413b6ff648e09a2f4ff371d588c9" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:19:49.546847 10340 tablet.cc:2369] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:49.549062 10505 mvcc.cc:205] Tried to move back new op lower bound from 7334082926731382784 to 7334082926563119104. Current Snapshot: MvccSnapshot[applied={T|T < 7334082926731382784}]
I20260927 22:19:49.565088 10014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:49.565397 10014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:49.569020 10014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07f455223f5b4e4cac8397b1e37aff90: Sending DeleteTablet for 3 replicas of tablet 90479c8cd05145e1ad30a630046a71d4
I20260927 22:19:49.570188 10407 tablet_service.cc:1514] Processing DeleteTablet for tablet 90479c8cd05145e1ad30a630046a71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:49 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:50142
I20260927 22:19:49.570231 10140 tablet_service.cc:1514] Processing DeleteTablet for tablet 90479c8cd05145e1ad30a630046a71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:49 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:37124
I20260927 22:19:49.570349 10274 tablet_service.cc:1514] Processing DeleteTablet for tablet 90479c8cd05145e1ad30a630046a71d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:49 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35160
I20260927 22:19:49.571316 10516 tablet_replica.cc:330] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: stopping tablet replica
I20260927 22:19:49.572674 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10077
I20260927 22:19:49.574085 10517 tablet_replica.cc:330] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: stopping tablet replica
I20260927 22:19:49.574640 10517 raft_consensus.cc:2229] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:49.575395 10517 raft_consensus.cc:2258] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:49.576232 10515 tablet_replica.cc:330] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: stopping tablet replica
I20260927 22:19:49.577304 10515 raft_consensus.cc:2229] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:49.571725 10516 raft_consensus.cc:2229] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:49.577901 10515 raft_consensus.cc:2258] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:49.578282 10516 raft_consensus.cc:2258] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:49.580405 10517 ts_tablet_manager.cc:1905] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:49.580411 10516 ts_tablet_manager.cc:1905] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:49.586618 10515 ts_tablet_manager.cc:1905] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:49.588058 10517 ts_tablet_manager.cc:1918] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:49.588217 10517 log.cc:1196] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/90479c8cd05145e1ad30a630046a71d4
I20260927 22:19:49.588748 10517 ts_tablet_manager.cc:1939] T 90479c8cd05145e1ad30a630046a71d4 P 0f727ef7d79b4f32af897c7b93083ec5: Deleting consensus metadata
I20260927 22:19:49.589548 10516 ts_tablet_manager.cc:1918] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:49.589715 10516 log.cc:1196] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/90479c8cd05145e1ad30a630046a71d4
I20260927 22:19:49.590279 10516 ts_tablet_manager.cc:1939] T 90479c8cd05145e1ad30a630046a71d4 P 4357a3ae5bb6404b82d7c8133bc50cea: Deleting consensus metadata
I20260927 22:19:49.590580 10003 catalog_manager.cc:4841] TS 0f727ef7d79b4f32af897c7b93083ec5 (127.23.173.66:39513): tablet 90479c8cd05145e1ad30a630046a71d4 (table test-table [id=5b4f9746b83c442ea2c5a33eb8491741]) successfully deleted
I20260927 22:19:49.591872 10003 catalog_manager.cc:4841] TS 4357a3ae5bb6404b82d7c8133bc50cea (127.23.173.65:45735): tablet 90479c8cd05145e1ad30a630046a71d4 (table test-table [id=5b4f9746b83c442ea2c5a33eb8491741]) successfully deleted
I20260927 22:19:49.593714 10515 ts_tablet_manager.cc:1918] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:49.593914 10515 log.cc:1196] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/90479c8cd05145e1ad30a630046a71d4
I20260927 22:19:49.594444 10515 ts_tablet_manager.cc:1939] T 90479c8cd05145e1ad30a630046a71d4 P ee7c413b6ff648e09a2f4ff371d588c9: Deleting consensus metadata
Sep 27 22:19:49 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:19:49.595783 10001 catalog_manager.cc:4841] TS ee7c413b6ff648e09a2f4ff371d588c9 (127.23.173.67:40469): tablet 90479c8cd05145e1ad30a630046a71d4 (table test-table [id=5b4f9746b83c442ea2c5a33eb8491741]) successfully deleted
I20260927 22:19:49.602123 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.573516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.23.173.65:45735)
0927 22:19:49.573754 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.573764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.579006 (+ 5242us) server_negotiation.cc:407] Connection header received
0927 22:19:49.579297 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.579302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.579389 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.579482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:49.581312 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.582561 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.584504 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.585287 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.585391 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.595861 (+ 10470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:49.595876 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:49.595894 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:49.595938 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:49.599115 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.599934 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.599941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.599943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.600078 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.600543 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.600546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.600548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.600896 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:49.601178 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.601580 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.601755 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:19:49.602835 10200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:49.701875 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10077
I20260927 22:19:49.721086 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10211
Sep 27 22:19:49 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:19:49.736903 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.721879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.23.173.66:39513)
0927 22:19:49.722093 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.722100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.722138 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:19:49.722504 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.722510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.722589 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.722700 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:49.724013 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.724774 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.726337 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.726738 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.726881 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.731320 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:49.731334 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:49.731361 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:49.731400 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:49.734376 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.734991 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.735020 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.735026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.735183 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.735559 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.735563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.735565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.735987 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:49.736290 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.736478 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.736643 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:19:49.737610 10334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:49.848104 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10211
I20260927 22:19:49.864980 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10344
Sep 27 22:19:49 dist-test-slave-3wc3 krb5kdc[9971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:19:49.881501 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:49.865802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.23.173.67:40469)
0927 22:19:49.866040 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:49.866046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:49.866066 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:19:49.866239 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:49.866243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:49.866309 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:49.866452 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:49.867807 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.868561 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.869880 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:49.870334 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:49.870478 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:49.874849 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:49.874872 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:49.874888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:49.874927 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:49.878830 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.879525 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.879532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.879535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.879671 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:49.879992 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:49.879995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:49.879997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:49.880330 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:49.880549 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:49.880891 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:19:49.881076 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:19:49.882229 10467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:49.972024 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10344
I20260927 22:19:49.989394 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 9984
I20260927 22:19:49.990604 10046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:50.107685 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 9984
2026-09-27T22:19:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:50Z Disabled control of system clock
I20260927 22:19:50.183861 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43589
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:50.512509 10526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:50.512813 10526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:50.512985 10526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:50.520514 10526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:50.520617 10526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:50.520720 10526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:50.520828 10526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:50.520910 10526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:50.529992 10526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:43589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:50.531230 10526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:50.532963 10526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:50.544809 10532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:50.544911 10531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:50.544888 10534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:50.545809 10526 server_base.cc:1034] running on GCE node
I20260927 22:19:50.546607 10526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:50.547922 10526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:50.549163 10526 hybrid_clock.cc:648] HybridClock initialized: now 1790547590549129 us; error 43 us; skew 500 ppm
I20260927 22:19:50.549551 10526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:50.551822 10526 webserver.cc:460] Webserver started at http://127.23.173.126:36069/ using document root <none> and password file <none>
I20260927 22:19:50.552526 10526 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:50.552660 10526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:50.552979 10526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:50.555837 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "b4987ceae9a344a4b9c79dd4acebcd34"
format_stamp: "Formatted at 2026-09-27 22:19:50 on dist-test-slave-3wc3"
server_key: "4b6f17af243583cbe092ed5336a4de08"
server_key_iv: "92a727f0403160116b3555aad6002e0e"
server_key_version: "encryptionkey@0"
I20260927 22:19:50.556627 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "b4987ceae9a344a4b9c79dd4acebcd34"
format_stamp: "Formatted at 2026-09-27 22:19:50 on dist-test-slave-3wc3"
server_key: "4b6f17af243583cbe092ed5336a4de08"
server_key_iv: "92a727f0403160116b3555aad6002e0e"
server_key_version: "encryptionkey@0"
I20260927 22:19:50.561728 10526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:19:50.565346 10540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:50.566776 10526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:50.566984 10526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "b4987ceae9a344a4b9c79dd4acebcd34"
format_stamp: "Formatted at 2026-09-27 22:19:50 on dist-test-slave-3wc3"
server_key: "4b6f17af243583cbe092ed5336a4de08"
server_key_iv: "92a727f0403160116b3555aad6002e0e"
server_key_version: "encryptionkey@0"
I20260927 22:19:50.567209 10526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:50.584648 10526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:50.585476 10526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:50.585764 10526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:50.604140 10526 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:43589
I20260927 22:19:50.604153 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:43589 every 8 connection(s)
I20260927 22:19:50.605476 10526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:50.609133 10592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:50.615829 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10526
I20260927 22:19:50.616015 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:50.616607 24245 external_mini_cluster.cc:1428] Setting key 61453d850e1fa9e1cab8c7791c8ef422
I20260927 22:19:50.620409 10592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: Bootstrap starting.
I20260927 22:19:50.623894 10592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:50.624940 10592 log.cc:824] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:50.628085 10592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: No bootstrap required, opened a new log
I20260927 22:19:50.631225 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:50.618702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.23.173.126:43589)
0927 22:19:50.619897 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:50.619909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:50.619955 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:19:50.621853 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:50.621873 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:50.622384 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:50.622682 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:50.623281 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:50.624876 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:50.626683 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:50.627375 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:50.627515 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:50.628809 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:50.628817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:50.628830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:50.629356 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:50.629442 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:50.629462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:50.629551 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:50.629804 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:19:50.630281 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":380,"threads_started":1}
I20260927 22:19:50.635938 10592 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } }
I20260927 22:19:50.636281 10592 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:50.636384 10592 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4987ceae9a344a4b9c79dd4acebcd34, State: Initialized, Role: FOLLOWER
I20260927 22:19:50.636945 10592 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } }
I20260927 22:19:50.637178 10592 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:50.637320 10592 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:50.637504 10592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:50.639729 10592 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } }
I20260927 22:19:50.640199 10592 leader_election.cc:302] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4987ceae9a344a4b9c79dd4acebcd34; no voters:
I20260927 22:19:50.640741 10592 leader_election.cc:288] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:50.640879 10597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:50.641675 10597 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [term 1 LEADER]: Becoming Leader. State: Replica: b4987ceae9a344a4b9c79dd4acebcd34, State: Running, Role: LEADER
I20260927 22:19:50.642215 10597 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } }
I20260927 22:19:50.642850 10592 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:50.645401 10598 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } } }
I20260927 22:19:50.645462 10599 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4987ceae9a344a4b9c79dd4acebcd34. Latest consensus state: current_term: 1 leader_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4987ceae9a344a4b9c79dd4acebcd34" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43589 } } }
I20260927 22:19:50.646575 10598 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:50.646632 10599 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:50.651269 10607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:50.656489 10607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:50.665261 10607 catalog_manager.cc:1347] Generated new cluster ID: 3525a5e26f1d4c50b1856c89e0a51f1d
I20260927 22:19:50.665400 10607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:50.677636 10607 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:50.678968 10607 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:50.690598 10607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: Generated new TSK 0
I20260927 22:19:50.691615 10607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:19:50.697500 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--builtin_ntp_servers=127.23.173.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:51.029425 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:51.029696 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:51.029840 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:51.037014 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:51.037106 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:51.037281 10616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:51.046567 10616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:51.047787 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:51.049615 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:51.063269 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.063351 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.063330 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:51.064126 10616 server_base.cc:1034] running on GCE node
I20260927 22:19:51.064697 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:51.065536 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:51.066744 10616 hybrid_clock.cc:648] HybridClock initialized: now 1790547591066702 us; error 28 us; skew 500 ppm
I20260927 22:19:51.067083 10616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:51.069265 10616 webserver.cc:460] Webserver started at http://127.23.173.65:36441/ using document root <none> and password file <none>
I20260927 22:19:51.069911 10616 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:51.070088 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:51.070447 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:51.073269 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "0ef89c37a1bb43bd909c00853579ba01"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "1df10969837ae05266921ce62e191c7a"
server_key_iv: "55db63def8f0625c11e6a8af0220d9ce"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.074091 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "0ef89c37a1bb43bd909c00853579ba01"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "1df10969837ae05266921ce62e191c7a"
server_key_iv: "55db63def8f0625c11e6a8af0220d9ce"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.078919 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:19:51.082195 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.083452 10616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:19:51.083624 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "0ef89c37a1bb43bd909c00853579ba01"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "1df10969837ae05266921ce62e191c7a"
server_key_iv: "55db63def8f0625c11e6a8af0220d9ce"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.083824 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:51.100239 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:51.101022 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:51.101320 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:51.102294 10616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:51.103715 10616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:51.103828 10616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.103955 10616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:51.104054 10616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.137079 10616 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:38413
I20260927 22:19:51.137135 10742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:38413 every 8 connection(s)
I20260927 22:19:51.138478 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:51.144511 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10616
I20260927 22:19:51.144685 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:51.145324 24245 external_mini_cluster.cc:1428] Setting key 37db2343a950ca784cb836cc04333650
I20260927 22:19:51.148788 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--builtin_ntp_servers=127.23.173.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:51.153127 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:51.141144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:43931 (local address 127.23.173.126:43589)
0927 22:19:51.141597 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:51.141603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:51.142928 (+ 1325us) server_negotiation.cc:407] Connection header received
0927 22:19:51.145066 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:51.145071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:51.145144 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:51.145245 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:51.146663 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.147790 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.150273 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.150690 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.150874 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:51.152358 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:51.152363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:51.152366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:51.152383 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:51.152419 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:51.152437 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:51.152539 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:51.152802 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:19:51.152874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":207,"threads_started":1}
I20260927 22:19:51.154644 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:51.141814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43589 (local address 127.23.173.65:43931)
0927 22:19:51.142770 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:51.142821 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:51.144764 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:51.145539 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:51.145551 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:51.145673 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:51.146458 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:51.146484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.147943 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.147948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:51.150134 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:51.150153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.150983 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.150994 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:51.151308 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:51.152028 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:51.152031 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:51.152069 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:51.152148 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:51.152571 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:51.152585 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:51.152597 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:19:51.152779 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 22:19:51.153195 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":326,"threads_started":1}
I20260927 22:19:51.156248 10743 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43589
I20260927 22:19:51.156570 10743 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:51.157428 10743 heartbeater.cc:511] Master 127.23.173.126:43589 requested a full tablet report, sending...
I20260927 22:19:51.159783 10557 ts_manager.cc:194] Registered new tserver with Master: 0ef89c37a1bb43bd909c00853579ba01 (127.23.173.65:38413)
I20260927 22:19:51.162005 10557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.65:43931
W20260927 22:19:51.479806 10747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:51.480079 10747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:51.480165 10747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:51.487341 10747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:51.487428 10747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:51.487581 10747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:51.496409 10747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:51.497541 10747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:51.499212 10747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:51.511914 10753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.511992 10752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.511883 10755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:51.512758 10747 server_base.cc:1034] running on GCE node
I20260927 22:19:51.513307 10747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:51.514217 10747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:51.515493 10747 hybrid_clock.cc:648] HybridClock initialized: now 1790547591515443 us; error 47 us; skew 500 ppm
I20260927 22:19:51.515815 10747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:51.518049 10747 webserver.cc:460] Webserver started at http://127.23.173.66:40979/ using document root <none> and password file <none>
I20260927 22:19:51.518715 10747 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:51.518834 10747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:51.519145 10747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:51.521872 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "6bd0b2018bd741b5a45df559f2f8fbcb"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "9e16b8791a3b91925137af146590a94b"
server_key_iv: "280b6889423fbc87a418f573ded64239"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.522718 10747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "6bd0b2018bd741b5a45df559f2f8fbcb"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "9e16b8791a3b91925137af146590a94b"
server_key_iv: "280b6889423fbc87a418f573ded64239"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.527644 10747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:19:51.530993 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.532215 10747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:51.532397 10747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "6bd0b2018bd741b5a45df559f2f8fbcb"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "9e16b8791a3b91925137af146590a94b"
server_key_iv: "280b6889423fbc87a418f573ded64239"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.532606 10747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:51.554029 10747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:51.554901 10747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:51.555213 10747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:51.556239 10747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:51.557757 10747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:51.557884 10747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.558040 10747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:51.558132 10747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.591593 10747 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:46181
I20260927 22:19:51.591678 10873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:46181 every 8 connection(s)
I20260927 22:19:51.592882 10747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:51.597167 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10747
I20260927 22:19:51.597347 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:51.597913 24245 external_mini_cluster.cc:1428] Setting key b43c92533011bbb87b1d853e4fba8361
I20260927 22:19:51.601007 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--builtin_ntp_servers=127.23.173.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:51.607682 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:51.595326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:38147 (local address 127.23.173.126:43589)
0927 22:19:51.595536 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:51.595542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:51.597206 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 22:19:51.599498 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:51.599504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:51.599579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:51.599707 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:51.601053 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.601845 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.604369 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.605024 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.605157 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:51.606822 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:51.606829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:51.606834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:51.606864 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:51.606902 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:51.606916 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:51.607002 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:51.607261 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:19:51.607359 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:19:51.609061 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:51.595940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43589 (local address 127.23.173.66:38147)
0927 22:19:51.597020 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:51.597075 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:51.599231 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:51.599952 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:51.599962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:51.600086 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:51.600875 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:51.600900 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.602053 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.602061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:51.604204 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:51.604222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:51.605268 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:51.605277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:51.605615 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:51.606457 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:51.606463 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:51.606517 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:51.606623 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:51.607026 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:51.607040 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:51.607051 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:19:51.607225 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 22:19:51.607687 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":317,"threads_started":1}
I20260927 22:19:51.610611 10874 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43589
I20260927 22:19:51.610934 10874 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:51.611685 10874 heartbeater.cc:511] Master 127.23.173.126:43589 requested a full tablet report, sending...
I20260927 22:19:51.613114 10557 ts_manager.cc:194] Registered new tserver with Master: 6bd0b2018bd741b5a45df559f2f8fbcb (127.23.173.66:46181)
I20260927 22:19:51.613932 10557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.66:38147
W20260927 22:19:51.935688 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:51.935998 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:51.936092 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:51.943226 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:51.943331 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:51.943444 10877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:51.952364 10877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:51.953611 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:51.955350 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:51.968441 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.968405 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:51.968508 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:51.969161 10877 server_base.cc:1034] running on GCE node
I20260927 22:19:51.969731 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:51.970803 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:51.972029 10877 hybrid_clock.cc:648] HybridClock initialized: now 1790547591971977 us; error 61 us; skew 500 ppm
I20260927 22:19:51.972344 10877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:51.974530 10877 webserver.cc:460] Webserver started at http://127.23.173.67:38035/ using document root <none> and password file <none>
I20260927 22:19:51.975211 10877 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:51.975345 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:51.975665 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:51.978621 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "133e0a03e639489db36d1afa5c5b2e8a"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "76fbcc93fb75ed31636a977215810f28"
server_key_iv: "6db1eeb64ca2400088ee6b2d5fb90b62"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.979387 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "133e0a03e639489db36d1afa5c5b2e8a"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "76fbcc93fb75ed31636a977215810f28"
server_key_iv: "6db1eeb64ca2400088ee6b2d5fb90b62"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.984229 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:19:51.987602 10891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:51.988911 10877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:19:51.989085 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "133e0a03e639489db36d1afa5c5b2e8a"
format_stamp: "Formatted at 2026-09-27 22:19:51 on dist-test-slave-3wc3"
server_key: "76fbcc93fb75ed31636a977215810f28"
server_key_iv: "6db1eeb64ca2400088ee6b2d5fb90b62"
server_key_version: "encryptionkey@0"
I20260927 22:19:51.989296 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:52.011221 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:52.012059 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:52.012375 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:52.013322 10877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:52.014814 10877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:52.014941 10877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:52.015080 10877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:52.015170 10877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:52.048461 10877 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:36665
I20260927 22:19:52.048552 11003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:36665 every 8 connection(s)
I20260927 22:19:52.049775 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:52.051764 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 10877
I20260927 22:19:52.051939 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:52.052591 24245 external_mini_cluster.cc:1428] Setting key 5cd1e6b9d15fc71b4940bd583fab2502
I20260927 22:19:52.064229 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.053155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:40263 (local address 127.23.173.126:43589)
0927 22:19:52.053382 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.053388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.055102 (+ 1714us) server_negotiation.cc:407] Connection header received
0927 22:19:52.057745 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.057749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.057873 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.058009 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:52.059310 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.060067 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.061682 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.062166 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.062264 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.063550 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:52.063555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:52.063558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:52.063575 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:52.063607 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:52.063618 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:52.063688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.063898 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.063980 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:19:52.065276 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.053595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43589 (local address 127.23.173.67:40263)
0927 22:19:52.054887 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.054963 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.057418 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.058312 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.058321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.058470 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:52.059142 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.059158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.060260 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.060264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.061537 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.061549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.062354 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.062359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.062576 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.063223 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:52.063225 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:52.063266 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:52.063367 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:52.063703 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:52.063710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:52.063716 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.063857 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.064154 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":389,"threads_started":1}
I20260927 22:19:52.066815 11004 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43589
I20260927 22:19:52.067092 11004 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:52.067739 11004 heartbeater.cc:511] Master 127.23.173.126:43589 requested a full tablet report, sending...
I20260927 22:19:52.069192 10557 ts_manager.cc:194] Registered new tserver with Master: 133e0a03e639489db36d1afa5c5b2e8a (127.23.173.67:36665)
I20260927 22:19:52.070405 10557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.67:40263
I20260927 22:19:52.070997 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:52.080695 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.075064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.23.173.126:43589)
0927 22:19:52.075218 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.075222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.075688 (+ 466us) server_negotiation.cc:407] Connection header received
0927 22:19:52.075820 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.075823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.075894 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.075986 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:52.076401 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.077163 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.078663 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.079034 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.079123 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.080105 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:52.080110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:52.080113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:52.080131 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:52.080169 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:52.080181 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:52.080287 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.080441 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.080483 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:19:52.084599 10557 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:52.091675 10557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:52.125932 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.103355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:46181 (local address 127.0.0.1:52908)
0927 22:19:52.115985 (+ 12630us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.116063 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.116286 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.117349 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.117362 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.117404 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:52.117994 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.118012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.120307 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.120317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.122588 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.122598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124901 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.124906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.125167 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.125224 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.125359 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.125575 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12275,"thread_start_us":12070,"threads_started":1}
I20260927 22:19:52.125983 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.103502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.23.173.66:46181)
0927 22:19:52.116373 (+ 12871us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.116384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.116422 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:19:52.116539 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.116547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.116816 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.117035 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:52.118273 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.120121 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.122832 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124335 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.124489 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.124766 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.125454 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.125637 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12698,"thread_start_us":12952,"threads_started":1}
I20260927 22:19:52.129282 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.104327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.23.173.65:38413)
0927 22:19:52.121125 (+ 16798us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.121134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.121168 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:19:52.121273 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.121281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.121536 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.121739 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:52.122468 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124047 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.127075 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.128380 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.128494 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.128663 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.128919 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.129042 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16630,"thread_start_us":16531,"threads_started":1}
I20260927 22:19:52.129333 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.103837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60624)
0927 22:19:52.120077 (+ 16240us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.120101 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.120222 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.121756 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.121761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.121787 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:52.122266 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.122279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124204 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.124210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.126847 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.126861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.128569 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.128575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.128731 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.128762 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.128845 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.128983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16049,"mutex_wait_us":121,"thread_start_us":15868,"threads_started":1}
I20260927 22:19:52.130448 10809 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 (DEFAULT_TABLE table=test-table [id=324c7a40a8cf480eb0d9dd4065c72f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:52.131937 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.103449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:36665 (local address 127.0.0.1:60986)
0927 22:19:52.115986 (+ 12537us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.116668 (+ 682us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.116774 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.122014 (+ 5240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.122023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.122055 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:52.122489 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.122502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124392 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.124397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.126916 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.126930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.130861 (+ 3931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.130869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.131081 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.131115 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.131199 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.131597 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12372,"mutex_wait_us":94,"thread_start_us":12210,"threads_started":1}
I20260927 22:19:52.131976 10809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:52.132879 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.104115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.23.173.67:36665)
0927 22:19:52.121160 (+ 17045us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.121167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.121196 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:19:52.121275 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.121281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.121536 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.121739 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:52.122727 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.124224 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.127139 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.130655 (+ 3516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.130798 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.131121 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.131292 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.132554 (+ 1262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16515,"thread_start_us":16363,"threads_started":1}
I20260927 22:19:52.133237 10678 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 (DEFAULT_TABLE table=test-table [id=324c7a40a8cf480eb0d9dd4065c72f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:52.134611 10678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:52.136643 10939 tablet_service.cc:1467] Processing CreateTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 (DEFAULT_TABLE table=test-table [id=324c7a40a8cf480eb0d9dd4065c72f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:52.138976 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:52.147943 11022 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Bootstrap starting.
I20260927 22:19:52.151736 11023 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Bootstrap starting.
I20260927 22:19:52.151924 11022 tablet_bootstrap.cc:654] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:52.153226 11022 log.cc:824] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:52.155550 11023 tablet_bootstrap.cc:654] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:52.156132 11022 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: No bootstrap required, opened a new log
I20260927 22:19:52.156443 11022 ts_tablet_manager.cc:1397] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 22:19:52.156911 11025 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Bootstrap starting.
I20260927 22:19:52.157382 11023 log.cc:824] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:52.161573 11025 tablet_bootstrap.cc:654] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:52.162947 11025 log.cc:824] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:52.164995 10743 heartbeater.cc:503] Master 127.23.173.126:43589 was elected leader, sending a full tablet report...
I20260927 22:19:52.166096 11023 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: No bootstrap required, opened a new log
I20260927 22:19:52.166396 11023 ts_tablet_manager.cc:1397] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260927 22:19:52.166975 11025 tablet_bootstrap.cc:492] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: No bootstrap required, opened a new log
I20260927 22:19:52.167292 11025 ts_tablet_manager.cc:1397] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 22:19:52.167224 11022 raft_consensus.cc:348] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.167706 11022 raft_consensus.cc:374] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:52.167816 11022 raft_consensus.cc:729] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd0b2018bd741b5a45df559f2f8fbcb, State: Initialized, Role: FOLLOWER
I20260927 22:19:52.168594 11022 consensus_queue.cc:260] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.171447 10874 heartbeater.cc:503] Master 127.23.173.126:43589 was elected leader, sending a full tablet report...
I20260927 22:19:52.172163 11022 ts_tablet_manager.cc:1428] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260927 22:19:52.177093 11023 raft_consensus.cc:348] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.177469 11023 raft_consensus.cc:374] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:52.177588 11023 raft_consensus.cc:729] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef89c37a1bb43bd909c00853579ba01, State: Initialized, Role: FOLLOWER
I20260927 22:19:52.178229 11023 consensus_queue.cc:260] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.178195 11025 raft_consensus.cc:348] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.178571 11025 raft_consensus.cc:374] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:52.178673 11025 raft_consensus.cc:729] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133e0a03e639489db36d1afa5c5b2e8a, State: Initialized, Role: FOLLOWER
I20260927 22:19:52.179270 11025 consensus_queue.cc:260] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.179999 11023 ts_tablet_manager.cc:1428] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260927 22:19:52.180919 11004 heartbeater.cc:503] Master 127.23.173.126:43589 was elected leader, sending a full tablet report...
I20260927 22:19:52.181610 11025 ts_tablet_manager.cc:1428] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260927 22:19:52.201756 11030 raft_consensus.cc:482] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:52.202127 11030 raft_consensus.cc:504] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.203889 11030 leader_election.cc:288] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd0b2018bd741b5a45df559f2f8fbcb (127.23.173.66:46181), 0ef89c37a1bb43bd909c00853579ba01 (127.23.173.65:38413)
I20260927 22:19:52.211104 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.204225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:45007 (local address 127.23.173.66:46181)
0927 22:19:52.204454 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.204461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.204875 (+ 414us) server_negotiation.cc:407] Connection header received
0927 22:19:52.204965 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.204971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.205068 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.205154 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:52.205618 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.206634 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.209079 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.210210 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.210361 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.210430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.210713 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.210843 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:19:52.211102 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.204273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:46181 (local address 127.23.173.67:45007)
0927 22:19:52.204580 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.204599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.204790 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.205205 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.205210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.205237 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:52.205469 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.205479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.206759 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.206762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.208902 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.208916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.210366 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.210371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.210537 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.210561 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.210634 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.210769 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":17}
I20260927 22:19:52.211684 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.204534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:53963 (local address 127.23.173.65:38413)
0927 22:19:52.204807 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.204816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.205238 (+ 422us) server_negotiation.cc:407] Connection header received
0927 22:19:52.205441 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.205445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.205516 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.205631 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:52.206172 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.207039 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.209416 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.210635 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.210784 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.210846 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.211332 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.211477 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:19:52.212195 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.204370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.23.173.67:53963)
0927 22:19:52.205103 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.205119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:52.205309 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:52.205648 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:52.205651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:52.205671 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:52.205949 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.206009 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.207176 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.207181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.209209 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:52.209219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.210797 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.210805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:52.210984 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.211011 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:19:52.211864 (+ 853us) client_negotiation.cc:240] Negotiation successful
0927 22:19:52.211966 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":28,"spinlock_wait_cycles":21248,"thread_start_us":433,"threads_started":1}
I20260927 22:19:52.212225 10829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfcfbe9eb7f4ccf844d3e6c496a3748" candidate_uuid: "133e0a03e639489db36d1afa5c5b2e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" is_pre_election: true
I20260927 22:19:52.212843 10829 raft_consensus.cc:2454] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133e0a03e639489db36d1afa5c5b2e8a in term 0.
I20260927 22:19:52.213271 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfcfbe9eb7f4ccf844d3e6c496a3748" candidate_uuid: "133e0a03e639489db36d1afa5c5b2e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ef89c37a1bb43bd909c00853579ba01" is_pre_election: true
I20260927 22:19:52.213586 10895 leader_election.cc:302] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133e0a03e639489db36d1afa5c5b2e8a, 6bd0b2018bd741b5a45df559f2f8fbcb; no voters:
I20260927 22:19:52.213783 10698 raft_consensus.cc:2454] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133e0a03e639489db36d1afa5c5b2e8a in term 0.
I20260927 22:19:52.214006 11030 raft_consensus.cc:2781] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:52.214146 11030 raft_consensus.cc:482] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:52.214249 11030 raft_consensus.cc:3037] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:52.216409 11030 raft_consensus.cc:504] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.217060 11030 leader_election.cc:288] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [CANDIDATE]: Term 1 election: Requested vote from peers 6bd0b2018bd741b5a45df559f2f8fbcb (127.23.173.66:46181), 0ef89c37a1bb43bd909c00853579ba01 (127.23.173.65:38413)
I20260927 22:19:52.217422 10829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfcfbe9eb7f4ccf844d3e6c496a3748" candidate_uuid: "133e0a03e639489db36d1afa5c5b2e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb"
I20260927 22:19:52.217552 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dfcfbe9eb7f4ccf844d3e6c496a3748" candidate_uuid: "133e0a03e639489db36d1afa5c5b2e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ef89c37a1bb43bd909c00853579ba01"
I20260927 22:19:52.217639 10829 raft_consensus.cc:3037] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:52.217767 10698 raft_consensus.cc:3037] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:52.220930 10698 raft_consensus.cc:2454] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133e0a03e639489db36d1afa5c5b2e8a in term 1.
I20260927 22:19:52.220937 10829 raft_consensus.cc:2454] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133e0a03e639489db36d1afa5c5b2e8a in term 1.
I20260927 22:19:52.221344 10894 leader_election.cc:302] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef89c37a1bb43bd909c00853579ba01, 133e0a03e639489db36d1afa5c5b2e8a; no voters:
I20260927 22:19:52.221700 11030 raft_consensus.cc:2781] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:52.222218 11030 raft_consensus.cc:686] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 1 LEADER]: Becoming Leader. State: Replica: 133e0a03e639489db36d1afa5c5b2e8a, State: Running, Role: LEADER
I20260927 22:19:52.222676 11030 consensus_queue.cc:237] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } }
I20260927 22:19:52.227061 10557 catalog_manager.cc:5495] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 133e0a03e639489db36d1afa5c5b2e8a (127.23.173.67). New cstate: current_term: 1 leader_uuid: "133e0a03e639489db36d1afa5c5b2e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133e0a03e639489db36d1afa5c5b2e8a" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 36665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:52.242370 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.236810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.23.173.67:36665)
0927 22:19:52.237016 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.237023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.237156 (+ 133us) server_negotiation.cc:407] Connection header received
0927 22:19:52.237402 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.237406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.237504 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.237654 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:52.238060 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.238779 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.240119 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.240537 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.240668 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.241273 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:52.241755 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:52.241943 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.242108 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.242147 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:19:52.251045 10698 raft_consensus.cc:1261] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 1 FOLLOWER]: Refusing update from remote peer 133e0a03e639489db36d1afa5c5b2e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:52.251049 10829 raft_consensus.cc:1261] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 1 FOLLOWER]: Refusing update from remote peer 133e0a03e639489db36d1afa5c5b2e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:52.252180 11032 consensus_queue.cc:1035] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd0b2018bd741b5a45df559f2f8fbcb" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:52.252584 11030 consensus_queue.cc:1035] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef89c37a1bb43bd909c00853579ba01" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:52.265193 11035 mvcc.cc:205] Tried to move back new op lower bound from 7334082937849344000 to 7334082937748127744. Current Snapshot: MvccSnapshot[applied={T|T < 7334082937849344000}]
I20260927 22:19:52.265609 11036 mvcc.cc:205] Tried to move back new op lower bound from 7334082937849344000 to 7334082937748127744. Current Snapshot: MvccSnapshot[applied={T|T < 7334082937849344000}]
I20260927 22:19:52.278815 10556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:52.279138 10556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:52.283283 10556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4987ceae9a344a4b9c79dd4acebcd34: Sending DeleteTablet for 3 replicas of tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748
I20260927 22:19:52.284477 10809 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:52 UTC) from {username='slave'} at 127.0.0.1:52908
I20260927 22:19:52.284511 10939 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:52 UTC) from {username='slave'} at 127.0.0.1:60986
I20260927 22:19:52.284876 10678 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:52 UTC) from {username='slave'} at 127.0.0.1:60624
I20260927 22:19:52.286783 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10616
I20260927 22:19:52.297073 11046 tablet_replica.cc:330] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: stopping tablet replica
I20260927 22:19:52.298024 11045 tablet_replica.cc:330] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: stopping tablet replica
I20260927 22:19:52.298357 11046 raft_consensus.cc:2229] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:52.298504 11045 raft_consensus.cc:2229] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:52.299021 11045 raft_consensus.cc:2258] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:52.299068 11046 raft_consensus.cc:2258] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:52.301316 11045 ts_tablet_manager.cc:1905] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:52.302580 11046 ts_tablet_manager.cc:1905] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:52.306890 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.288064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.23.173.65:38413)
0927 22:19:52.288247 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.288254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.297269 (+ 9015us) server_negotiation.cc:407] Connection header received
0927 22:19:52.297529 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.297535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.297652 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.297760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:52.298420 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.299712 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.302062 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.302722 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.302826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.304908 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:52.304921 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:52.304940 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:52.305700 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:52.305762 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:52.305786 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:52.305883 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.306380 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.306495 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:19:52.308422 11045 ts_tablet_manager.cc:1918] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:52.308537 10738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:52.308580 11045 log.cc:1196] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/0dfcfbe9eb7f4ccf844d3e6c496a3748
I20260927 22:19:52.309150 11045 ts_tablet_manager.cc:1939] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 6bd0b2018bd741b5a45df559f2f8fbcb: Deleting consensus metadata
I20260927 22:19:52.310883 11044 tablet_replica.cc:330] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: stopping tablet replica
I20260927 22:19:52.311329 11044 raft_consensus.cc:2229] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:52.312024 10544 catalog_manager.cc:4841] TS 6bd0b2018bd741b5a45df559f2f8fbcb (127.23.173.66:46181): tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 (table test-table [id=324c7a40a8cf480eb0d9dd4065c72f65]) successfully deleted
I20260927 22:19:52.312103 11044 raft_consensus.cc:2258] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:52.314432 11044 ts_tablet_manager.cc:1905] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:52.319703 11044 ts_tablet_manager.cc:1918] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:52.319850 11044 log.cc:1196] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/0dfcfbe9eb7f4ccf844d3e6c496a3748
I20260927 22:19:52.320320 11044 ts_tablet_manager.cc:1939] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 133e0a03e639489db36d1afa5c5b2e8a: Deleting consensus metadata
I20260927 22:19:52.321847 10541 catalog_manager.cc:4841] TS 133e0a03e639489db36d1afa5c5b2e8a (127.23.173.67:36665): tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748 (table test-table [id=324c7a40a8cf480eb0d9dd4065c72f65]) successfully deleted
I20260927 22:19:52.434669 11046 ts_tablet_manager.cc:1918] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:52.434965 11046 log.cc:1196] T 0dfcfbe9eb7f4ccf844d3e6c496a3748 P 0ef89c37a1bb43bd909c00853579ba01: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/0dfcfbe9eb7f4ccf844d3e6c496a3748
I20260927 22:19:52.435297 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10616
W20260927 22:19:52.453353 10543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.65:38413 (error 108)
I20260927 22:19:52.454161 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10747
I20260927 22:19:52.455644 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.455111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60648)
0927 22:19:52.455354 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.455489 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260927 22:19:52.455920 10543 catalog_manager.cc:4584] TS 0ef89c37a1bb43bd909c00853579ba01 (127.23.173.65:38413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dfcfbe9eb7f4ccf844d3e6c496a3748: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
I20260927 22:19:52.462375 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.455093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.23.173.66:46181)
0927 22:19:52.455282 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.455289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.455309 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:19:52.455526 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.455532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.455647 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.455752 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:52.456357 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.457148 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.459192 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.459671 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.459772 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.460905 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:52.460914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:52.460928 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:52.461528 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:52.461575 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:52.461594 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:52.461687 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.461837 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.461929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:19:52.463099 10869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:52.509836 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.509252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60650)
0927 22:19:52.509513 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.509625 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:19:52.510985 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.510583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60664)
0927 22:19:52.510748 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.510832 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:19:52.552098 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10747
I20260927 22:19:52.570039 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10877
I20260927 22:19:52.571692 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.571069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60672)
0927 22:19:52.571376 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.571473 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 22:19:52.572933 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.572504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60680)
0927 22:19:52.572691 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.572782 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:19:52.578227 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.571020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.23.173.67:36665)
0927 22:19:52.571208 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:52.571214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:52.571291 (+ 77us) server_negotiation.cc:407] Connection header received
0927 22:19:52.571517 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:52.571527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:52.571672 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:52.571826 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:52.572320 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.573417 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.574768 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:52.575235 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:52.575406 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:52.576575 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:52.576584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:52.576601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:52.577237 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:52.577288 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:52.577311 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:52.577456 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:52.577630 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 22:19:52.577721 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:19:52.579011 10999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:52.653476 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.652973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60690)
0927 22:19:52.653183 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.653289 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:19:52.654665 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.654275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60698)
0927 22:19:52.654480 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.654553 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 22:19:52.702443 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10877
I20260927 22:19:52.722667 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 10526
I20260927 22:19:52.723816 10587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:52.836480 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:52.835795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:38413 (local address 127.0.0.1:60706)
0927 22:19:52.836049 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:52.836198 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:38413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:19:52.836979 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 10526
2026-09-27T22:19:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2728 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:19:52 dist-test-slave-3wc3 krb5kdc[11055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:19:52 dist-test-slave-3wc3 krb5kdc[11055](info): set up 2 sockets
Sep 27 22:19:52 dist-test-slave-3wc3 krb5kdc[11055](info): commencing operation
krb5kdc: starting...
W20260927 22:19:53.946267 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:19:54 dist-test-slave-3wc3 krb5kdc[11055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:19:54.195319 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43507
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:54.530220 11068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:54.530586 11068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:54.530774 11068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:54.538511 11068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:54.538638 11068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:54.538723 11068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:54.538849 11068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:54.538931 11068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:54.549984 11068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:54.551409 11068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:54.552793 11068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:54.565732 11073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:54.565789 11074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:54.565697 11076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:54.566642 11068 server_base.cc:1034] running on GCE node
I20260927 22:19:54.567423 11068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:54.568817 11068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:54.570062 11068 hybrid_clock.cc:648] HybridClock initialized: now 1790547594569990 us; error 58 us; skew 500 ppm
I20260927 22:19:54.570413 11068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:54 dist-test-slave-3wc3 krb5kdc[11055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:54.581619 11068 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:19:54.583254 11068 webserver.cc:460] Webserver started at http://127.23.173.126:43279/ using document root <none> and password file <none>
I20260927 22:19:54.583964 11068 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:54.584103 11068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:54.584421 11068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:54.587396 11068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "3dc21bc396ea4c8cacd5e94868acccfc"
format_stamp: "Formatted at 2026-09-27 22:19:54 on dist-test-slave-3wc3"
server_key: "829b3b0482f8e46ffea3a2d3c87b3796"
server_key_iv: "99efa45337b269bc23a481a7a7967ee7"
server_key_version: "encryptionkey@0"
I20260927 22:19:54.588182 11068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "3dc21bc396ea4c8cacd5e94868acccfc"
format_stamp: "Formatted at 2026-09-27 22:19:54 on dist-test-slave-3wc3"
server_key: "829b3b0482f8e46ffea3a2d3c87b3796"
server_key_iv: "99efa45337b269bc23a481a7a7967ee7"
server_key_version: "encryptionkey@0"
I20260927 22:19:54.593103 11068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:19:54.596514 11083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:54.597834 11068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:19:54.598071 11068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "3dc21bc396ea4c8cacd5e94868acccfc"
format_stamp: "Formatted at 2026-09-27 22:19:54 on dist-test-slave-3wc3"
server_key: "829b3b0482f8e46ffea3a2d3c87b3796"
server_key_iv: "99efa45337b269bc23a481a7a7967ee7"
server_key_version: "encryptionkey@0"
I20260927 22:19:54.598276 11068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:54.615644 11068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:54.619971 11068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:54.620278 11068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:54.639246 11068 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:43507
I20260927 22:19:54.639266 11134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:43507 every 8 connection(s)
I20260927 22:19:54.640690 11068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:54.644490 11135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:54.650699 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11068
I20260927 22:19:54.650885 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:54.651360 24245 external_mini_cluster.cc:1428] Setting key a8b1112ea8d2ce45d48988f9e2511dbc
I20260927 22:19:54.655072 11135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: Bootstrap starting.
I20260927 22:19:54.659363 11135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:54.660543 11135 log.cc:824] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:54.663763 11135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: No bootstrap required, opened a new log
Sep 27 22:19:54 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:54.675603 11135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } }
I20260927 22:19:54.676074 11135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:54.676201 11135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc21bc396ea4c8cacd5e94868acccfc, State: Initialized, Role: FOLLOWER
I20260927 22:19:54.676936 11135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } }
I20260927 22:19:54.677193 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:54.653114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.23.173.126:43507)
0927 22:19:54.653866 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:54.653877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:54.653916 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:19:54.655863 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:54.655884 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:54.656433 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:54.656781 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:54.658863 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:54.660236 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:54.662093 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:54.662741 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:54.662861 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:54.667908 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:54.667925 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:54.667945 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:54.668002 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:54.672320 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:54.672969 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:54.672978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:54.672988 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:54.673217 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:54.673707 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:54.673714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:54.673718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:54.674547 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:54.674868 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:54.675179 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:19:54.675652 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":363,"threads_started":1}
I20260927 22:19:54.677268 11135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:54.677415 11135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:54.677656 11135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:54.680245 11135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } }
I20260927 22:19:54.680768 11135 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc21bc396ea4c8cacd5e94868acccfc; no voters:
I20260927 22:19:54.681351 11135 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:54.681457 11140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:54.682415 11140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [term 1 LEADER]: Becoming Leader. State: Replica: 3dc21bc396ea4c8cacd5e94868acccfc, State: Running, Role: LEADER
I20260927 22:19:54.682948 11140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } }
I20260927 22:19:54.683625 11135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:54.685496 11141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } } }
I20260927 22:19:54.686123 11141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:54.685745 11142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc21bc396ea4c8cacd5e94868acccfc. Latest consensus state: current_term: 1 leader_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc21bc396ea4c8cacd5e94868acccfc" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43507 } } }
I20260927 22:19:54.687090 11142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:54.688300 11146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:54.693439 11146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:54.703186 11146 catalog_manager.cc:1347] Generated new cluster ID: edf5211732dd49ec83677f1e56370ef5
I20260927 22:19:54.703291 11146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:54.717350 11146 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:54.718722 11146 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:54.730165 11146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: Generated new TSK 0
I20260927 22:19:54.730983 11146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.173.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
Entry for principal kudu/127.23.173.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab.
I20260927 22:19:54.802191 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:55.136569 11161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:55.136859 11161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:55.137022 11161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:55.144337 11161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:55.144448 11161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:55.144618 11161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:55.154554 11161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.65.keytab
--principal=kudu/127.23.173.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:55.155814 11161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:55.157073 11161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:55.170910 11166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:55.171000 11167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:55.170964 11169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:55.171856 11161 server_base.cc:1034] running on GCE node
I20260927 22:19:55.172425 11161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:55.173412 11161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:55.174651 11161 hybrid_clock.cc:648] HybridClock initialized: now 1790547595174590 us; error 56 us; skew 500 ppm
I20260927 22:19:55.174923 11161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:55 dist-test-slave-3wc3 krb5kdc[11055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:55.178966 11161 init.cc:377] Logged in from keytab as kudu/127.23.173.65@KRBTEST.COM (short username kudu)
I20260927 22:19:55.180451 11161 webserver.cc:460] Webserver started at http://127.23.173.65:44029/ using document root <none> and password file <none>
I20260927 22:19:55.181182 11161 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:55.181339 11161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:55.181701 11161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:55.185236 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "ff313e511b894013a2b6a47aa442afa9"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "fa3ccb8bdf776ac5b6d49a27d6fc8486"
server_key_iv: "ea7969648b0ef97fa45af8c448473b55"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.186137 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "ff313e511b894013a2b6a47aa442afa9"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "fa3ccb8bdf776ac5b6d49a27d6fc8486"
server_key_iv: "ea7969648b0ef97fa45af8c448473b55"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.191684 11161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:19:55.195328 11176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.196730 11161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:55.196918 11161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "ff313e511b894013a2b6a47aa442afa9"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "fa3ccb8bdf776ac5b6d49a27d6fc8486"
server_key_iv: "ea7969648b0ef97fa45af8c448473b55"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.197124 11161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:55.227692 11161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:55.232371 11161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:55.232744 11161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:55.233743 11161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:55.235199 11161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:55.235280 11161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.235481 11161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:55.235540 11161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.270885 11161 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:37081
I20260927 22:19:55.270929 11288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:37081 every 8 connection(s)
I20260927 22:19:55.272298 11161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:55.279806 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11161
I20260927 22:19:55.279995 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:55.280519 24245 external_mini_cluster.cc:1428] Setting key d016e1a1f55d40ef9cfeb00dfcd6aeac
Sep 27 22:19:55 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.65@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:55.294533 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:55.275060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:48267 (local address 127.23.173.126:43507)
0927 22:19:55.275528 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:55.275532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:55.276435 (+ 903us) server_negotiation.cc:407] Connection header received
0927 22:19:55.278370 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:55.278375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:55.278468 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:55.278597 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:55.280360 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.281312 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.283440 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.283936 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.284035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:55.288319 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:55.288351 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:55.288359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:55.288403 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:55.291494 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:55.292231 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:55.292241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:55.292243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:55.292359 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:55.292795 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:55.292804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:55.292809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:55.293082 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:55.293240 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:55.293911 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 22:19:55.294175 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":252,"threads_started":1}
I20260927 22:19:55.295403 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:55.275577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43507 (local address 127.23.173.65:48267)
0927 22:19:55.276268 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:55.276309 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:55.278115 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:55.278820 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:55.278828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:55.279332 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:55.280161 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:55.280185 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.281469 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.281488 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:55.283271 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:55.283293 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.284158 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.284166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:55.284503 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:55.285320 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:55.285354 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:55.288095 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:55.291642 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:55.291651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:55.291674 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:55.292096 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:55.292480 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:55.292485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:55.292490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:55.292672 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:55.293214 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:55.293222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:55.293540 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 22:19:55.293797 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 22:19:55.294322 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":283,"threads_started":1}
I20260927 22:19:55.296886 11289 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43507
I20260927 22:19:55.297214 11289 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:55.298084 11289 heartbeater.cc:511] Master 127.23.173.126:43507 requested a full tablet report, sending...
I20260927 22:19:55.300253 11100 ts_manager.cc:194] Registered new tserver with Master: ff313e511b894013a2b6a47aa442afa9 (127.23.173.65:37081)
I20260927 22:19:55.301926 11100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.65@KRBTEST.COM'} at 127.23.173.65:48267
WARNING: no policy specified for kudu/127.23.173.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
Entry for principal kudu/127.23.173.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab.
I20260927 22:19:55.339020 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:55.663056 11295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:55.663324 11295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:55.663563 11295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:55.672256 11295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:55.672374 11295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:55.672577 11295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:55.683526 11295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.66.keytab
--principal=kudu/127.23.173.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:55.684846 11295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:55.686251 11295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:55.699437 11300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:55.699465 11301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:55.699749 11303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:55.700682 11295 server_base.cc:1034] running on GCE node
I20260927 22:19:55.701376 11295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:55.702559 11295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:55.703814 11295 hybrid_clock.cc:648] HybridClock initialized: now 1790547595703771 us; error 32 us; skew 500 ppm
I20260927 22:19:55.704155 11295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:55 dist-test-slave-3wc3 krb5kdc[11055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:55.707305 11295 init.cc:377] Logged in from keytab as kudu/127.23.173.66@KRBTEST.COM (short username kudu)
I20260927 22:19:55.708655 11295 webserver.cc:460] Webserver started at http://127.23.173.66:44213/ using document root <none> and password file <none>
I20260927 22:19:55.709277 11295 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:55.709373 11295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:55.709642 11295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:55.712561 11295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "13511b6d4c754365b6b84a081b41657a"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "121f5963fc4c833b04b9d6617fec1a9a"
server_key_iv: "0a9ae88dd1e3d4b13da41e8ef1ac3f3a"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.713269 11295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "13511b6d4c754365b6b84a081b41657a"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "121f5963fc4c833b04b9d6617fec1a9a"
server_key_iv: "0a9ae88dd1e3d4b13da41e8ef1ac3f3a"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.718554 11295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:19:55.722049 11310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.723394 11295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:19:55.723569 11295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "13511b6d4c754365b6b84a081b41657a"
format_stamp: "Formatted at 2026-09-27 22:19:55 on dist-test-slave-3wc3"
server_key: "121f5963fc4c833b04b9d6617fec1a9a"
server_key_iv: "0a9ae88dd1e3d4b13da41e8ef1ac3f3a"
server_key_version: "encryptionkey@0"
I20260927 22:19:55.723732 11295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:55.750476 11295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:55.755220 11295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:55.755537 11295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:55.756502 11295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:55.757798 11295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:55.757884 11295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.758064 11295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:55.758116 11295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:55.791245 11295 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:42169
I20260927 22:19:55.791292 11422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:42169 every 8 connection(s)
I20260927 22:19:55.792536 11295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:55.802428 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11295
I20260927 22:19:55.802608 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:55.803139 24245 external_mini_cluster.cc:1428] Setting key 38357349d666a9112e93fc4b55c630b0
Sep 27 22:19:55 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547595, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.66@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:55.813022 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:55.795189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:36617 (local address 127.23.173.126:43507)
0927 22:19:55.795663 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:55.795667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:55.796619 (+ 952us) server_negotiation.cc:407] Connection header received
0927 22:19:55.798514 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:55.798520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:55.798614 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:55.798739 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:55.800211 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.800917 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.802922 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.803439 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.803582 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:55.807269 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:55.807275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:55.807280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:55.807321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:55.809935 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:55.810701 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:55.810706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:55.810710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:55.810847 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:55.811281 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:55.811286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:55.811289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:55.811555 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:55.811715 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:55.812408 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 22:19:55.812646 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":275,"threads_started":1}
I20260927 22:19:55.813949 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:55.795674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43507 (local address 127.23.173.66:36617)
0927 22:19:55.796463 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:55.796509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:55.798241 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:55.798972 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:55.798980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:55.799479 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:55.800036 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:55.800053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.801089 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.801093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:55.802761 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:55.802784 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:55.803710 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:55.803720 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:55.803949 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:55.804634 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:55.804656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:55.807047 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:55.810134 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:55.810142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:55.810158 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:55.810571 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:55.810968 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:55.810973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:55.810977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:55.811157 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:55.811683 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:55.811691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:55.812010 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 22:19:55.812277 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:19:55.812812 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":275,"threads_started":1}
I20260927 22:19:55.815400 11423 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43507
I20260927 22:19:55.815727 11423 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:55.816592 11423 heartbeater.cc:511] Master 127.23.173.126:43507 requested a full tablet report, sending...
I20260927 22:19:55.818166 11100 ts_manager.cc:194] Registered new tserver with Master: 13511b6d4c754365b6b84a081b41657a (127.23.173.66:42169)
I20260927 22:19:55.819428 11100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.66@KRBTEST.COM'} at 127.23.173.66:36617
WARNING: no policy specified for kudu/127.23.173.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
Entry for principal kudu/127.23.173.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab.
I20260927 22:19:55.861167 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:19:56.185776 11429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:56.186123 11429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:56.186296 11429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:56.193427 11429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:56.193573 11429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:56.193738 11429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:56.203495 11429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.67.keytab
--principal=kudu/127.23.173.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:56.204691 11429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:56.206004 11429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:56.218971 11434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:56.219056 11437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:56.219074 11435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:56.219667 11429 server_base.cc:1034] running on GCE node
I20260927 22:19:56.220252 11429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:56.221230 11429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:56.222445 11429 hybrid_clock.cc:648] HybridClock initialized: now 1790547596222388 us; error 64 us; skew 500 ppm
I20260927 22:19:56.222741 11429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:19:56 dist-test-slave-3wc3 krb5kdc[11055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547596, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:19:56.225682 11429 init.cc:377] Logged in from keytab as kudu/127.23.173.67@KRBTEST.COM (short username kudu)
I20260927 22:19:56.227098 11429 webserver.cc:460] Webserver started at http://127.23.173.67:39559/ using document root <none> and password file <none>
I20260927 22:19:56.227785 11429 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:56.227887 11429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:56.228191 11429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:56.230999 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "a33dff89a47d4b14b8cfdefde211c237"
format_stamp: "Formatted at 2026-09-27 22:19:56 on dist-test-slave-3wc3"
server_key: "2d18a04c576b24557b7a87f86f7d65af"
server_key_iv: "b5abaa9eb5f142e5ccd4169453286192"
server_key_version: "encryptionkey@0"
I20260927 22:19:56.231674 11429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "a33dff89a47d4b14b8cfdefde211c237"
format_stamp: "Formatted at 2026-09-27 22:19:56 on dist-test-slave-3wc3"
server_key: "2d18a04c576b24557b7a87f86f7d65af"
server_key_iv: "b5abaa9eb5f142e5ccd4169453286192"
server_key_version: "encryptionkey@0"
I20260927 22:19:56.236340 11429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:19:56.239444 11444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:56.240619 11429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:19:56.240741 11429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "a33dff89a47d4b14b8cfdefde211c237"
format_stamp: "Formatted at 2026-09-27 22:19:56 on dist-test-slave-3wc3"
server_key: "2d18a04c576b24557b7a87f86f7d65af"
server_key_iv: "b5abaa9eb5f142e5ccd4169453286192"
server_key_version: "encryptionkey@0"
I20260927 22:19:56.240898 11429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:56.262578 11429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:56.266338 11429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:56.266592 11429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:56.267494 11429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:56.268718 11429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:56.268786 11429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:56.268865 11429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:56.268908 11429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:56.302387 11429 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:43915
I20260927 22:19:56.302474 11556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:43915 every 8 connection(s)
I20260927 22:19:56.303637 11429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:56.304639 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11429
I20260927 22:19:56.304821 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:56.305405 24245 external_mini_cluster.cc:1428] Setting key 07328a667d410e7f5150add245574f85
I20260927 22:19:56.305796 11289 heartbeater.cc:503] Master 127.23.173.126:43507 was elected leader, sending a full tablet report...
Sep 27 22:19:56 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547596, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.67@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:19:56.324083 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.306662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:54323 (local address 127.23.173.126:43507)
0927 22:19:56.306838 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.306843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.308245 (+ 1402us) server_negotiation.cc:407] Connection header received
0927 22:19:56.310278 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.310283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.310380 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.310494 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:56.311947 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.312697 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.314297 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.314686 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.314820 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.318082 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:56.318086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:56.318090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:56.318122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:56.321361 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:56.322078 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:56.322082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:56.322084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:56.322197 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:56.322567 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:56.322570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:56.322572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:56.322833 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:56.322957 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.323647 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.323835 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:19:56.324944 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.307070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:43507 (local address 127.23.173.67:54323)
0927 22:19:56.308094 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.308130 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.310015 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.310731 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.310739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.311230 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:56.311777 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.311795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.312838 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.312844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.314147 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.314160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.314915 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.314922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.315145 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.315723 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:19:56.315743 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:56.317831 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:56.321497 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:56.321503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:56.321515 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:56.321912 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:56.322291 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:19:56.322295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:19:56.322297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:19:56.322464 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:19:56.322941 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:56.322946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:56.323251 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.323483 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.323832 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":623,"threads_started":1}
I20260927 22:19:56.326431 11557 heartbeater.cc:347] Connected to a master server at 127.23.173.126:43507
I20260927 22:19:56.326695 11557 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:56.327373 11557 heartbeater.cc:511] Master 127.23.173.126:43507 requested a full tablet report, sending...
I20260927 22:19:56.328794 11099 ts_manager.cc:194] Registered new tserver with Master: a33dff89a47d4b14b8cfdefde211c237 (127.23.173.67:43915)
I20260927 22:19:56.329660 11099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.173.67@KRBTEST.COM'} at 127.23.173.67:54323
I20260927 22:19:56.333097 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:56.349579 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.337175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.23.173.126:43507)
0927 22:19:56.337392 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.337401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.337712 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:19:56.337907 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.337913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.338049 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.338152 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:56.339532 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.340222 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.341512 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.341922 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.342087 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.344334 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:56.344339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:56.344342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:56.344370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:56.346845 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:56.347511 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:56.347516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:56.347519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:56.347669 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:56.348098 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:56.348101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:56.348103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:56.348381 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:56.348580 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.349062 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.349211 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:19:56.354048 11099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:56.363940 11099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:56.397977 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.375740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:43915 (local address 127.0.0.1:36882)
0927 22:19:56.389248 (+ 13508us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.389301 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.389997 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.391112 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.391120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.391136 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:56.391823 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.391835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393690 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.393695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.395643 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.395655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.397099 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.397105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.397233 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.397268 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.397386 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.397564 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13141,"mutex_wait_us":97,"thread_start_us":13077,"threads_started":1}
I20260927 22:19:56.398036 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.376248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.23.173.67:43915)
0927 22:19:56.377406 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.377420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.389473 (+ 12053us) server_negotiation.cc:407] Connection header received
0927 22:19:56.390296 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.390333 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.390615 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.390781 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:56.392012 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393549 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.395800 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.396924 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.397064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.397302 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.397458 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.397625 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":653,"threads_started":1}
I20260927 22:19:56.400928 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.378365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42169 (local address 127.0.0.1:35216)
0927 22:19:56.390270 (+ 11905us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.390288 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.390467 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.391028 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.391035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.391064 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:56.391580 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.391590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393544 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.393550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.396404 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.396419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.400025 (+ 3606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.400034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.400214 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.400249 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.400358 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.400477 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11752,"mutex_wait_us":249,"thread_start_us":11540,"threads_started":1}
I20260927 22:19:56.401005 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.378578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.23.173.66:42169)
0927 22:19:56.389524 (+ 10946us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.389533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.390373 (+ 840us) server_negotiation.cc:407] Connection header received
0927 22:19:56.390628 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.390641 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.390811 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.391015 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:56.391780 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393389 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.398684 (+ 5295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.399803 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.399931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.400054 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.400429 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.400669 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10779,"thread_start_us":10529,"threads_started":1}
I20260927 22:19:56.401284 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.376256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.23.173.65:37081)
0927 22:19:56.377939 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.377946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.390077 (+ 12131us) server_negotiation.cc:407] Connection header received
0927 22:19:56.390438 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.390445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.390613 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.390808 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:56.392247 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393559 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.396569 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.398067 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.398212 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.398456 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.400543 (+ 2087us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.400713 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":1364,"threads_started":1}
I20260927 22:19:56.402020 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.376007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:37081 (local address 127.0.0.1:59302)
0927 22:19:56.389922 (+ 13915us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.389944 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.390229 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.391093 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.391098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.391127 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:56.391835 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.391854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.393732 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.393738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.396404 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.396419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.400173 (+ 3754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.400178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.400300 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.400364 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.401490 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.401601 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13678,"mutex_wait_us":95,"thread_start_us":13510,"threads_started":1}
I20260927 22:19:56.402186 11492 tablet_service.cc:1467] Processing CreateTablet for tablet c3de48a77cc648628c7b1b55390e49e5 (DEFAULT_TABLE table=test-table [id=7a02a8f7f840499ca09967cbc7e712fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:56.403549 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de48a77cc648628c7b1b55390e49e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:56.408537 11224 tablet_service.cc:1467] Processing CreateTablet for tablet c3de48a77cc648628c7b1b55390e49e5 (DEFAULT_TABLE table=test-table [id=7a02a8f7f840499ca09967cbc7e712fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:56.409751 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de48a77cc648628c7b1b55390e49e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:56.409667 11358 tablet_service.cc:1467] Processing CreateTablet for tablet c3de48a77cc648628c7b1b55390e49e5 (DEFAULT_TABLE table=test-table [id=7a02a8f7f840499ca09967cbc7e712fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:56.410969 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3de48a77cc648628c7b1b55390e49e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:56.418120 11575 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Bootstrap starting.
I20260927 22:19:56.421767 11575 tablet_bootstrap.cc:654] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:56.422077 11576 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Bootstrap starting.
I20260927 22:19:56.423158 11575 log.cc:824] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:56.424288 11577 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Bootstrap starting.
I20260927 22:19:56.425565 11576 tablet_bootstrap.cc:654] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:56.426692 11576 log.cc:824] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:56.427713 11577 tablet_bootstrap.cc:654] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:56.427953 11575 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: No bootstrap required, opened a new log
I20260927 22:19:56.428247 11575 ts_tablet_manager.cc:1397] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:19:56.428874 11577 log.cc:824] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:56.429621 11576 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: No bootstrap required, opened a new log
I20260927 22:19:56.429883 11576 ts_tablet_manager.cc:1397] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:19:56.431748 11577 tablet_bootstrap.cc:492] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: No bootstrap required, opened a new log
I20260927 22:19:56.431990 11577 ts_tablet_manager.cc:1397] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:19:56.436981 11576 raft_consensus.cc:348] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.437319 11576 raft_consensus.cc:374] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:56.437451 11576 raft_consensus.cc:729] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff313e511b894013a2b6a47aa442afa9, State: Initialized, Role: FOLLOWER
I20260927 22:19:56.438103 11576 consensus_queue.cc:260] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.439349 11575 raft_consensus.cc:348] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.439720 11575 raft_consensus.cc:374] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:56.439814 11575 raft_consensus.cc:729] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33dff89a47d4b14b8cfdefde211c237, State: Initialized, Role: FOLLOWER
I20260927 22:19:56.440351 11575 consensus_queue.cc:260] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.441562 11557 heartbeater.cc:503] Master 127.23.173.126:43507 was elected leader, sending a full tablet report...
I20260927 22:19:56.442282 11577 raft_consensus.cc:348] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.442693 11577 raft_consensus.cc:374] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:56.442796 11577 raft_consensus.cc:729] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13511b6d4c754365b6b84a081b41657a, State: Initialized, Role: FOLLOWER
I20260927 22:19:56.443305 11577 consensus_queue.cc:260] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.443598 11575 ts_tablet_manager.cc:1428] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260927 22:19:56.444489 11423 heartbeater.cc:503] Master 127.23.173.126:43507 was elected leader, sending a full tablet report...
I20260927 22:19:56.445145 11577 ts_tablet_manager.cc:1428] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260927 22:19:56.446799 11576 ts_tablet_manager.cc:1428] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Time spent starting tablet: real 0.017s user 0.006s sys 0.007s
W20260927 22:19:56.524871 11290 tablet.cc:2369] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:56.544564 11424 tablet.cc:2369] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:56.555279 11558 tablet.cc:2369] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:56.807526 11581 raft_consensus.cc:482] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:56.807878 11581 raft_consensus.cc:504] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.809581 11581 leader_election.cc:288] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33dff89a47d4b14b8cfdefde211c237 (127.23.173.67:43915), 13511b6d4c754365b6b84a081b41657a (127.23.173.66:42169)
I20260927 22:19:56.816725 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.810107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:56031 (local address 127.23.173.66:42169)
0927 22:19:56.810339 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.810348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.811263 (+ 915us) server_negotiation.cc:407] Connection header received
0927 22:19:56.811477 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.811486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.811540 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.811652 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:56.812272 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.812951 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.814935 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.815743 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.815919 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.816003 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.816341 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.816463 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:19:56.816726 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.810114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:40843 (local address 127.23.173.67:43915)
0927 22:19:56.810317 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.810324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.811226 (+ 902us) server_negotiation.cc:407] Connection header received
0927 22:19:56.811462 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.811466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.811526 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.811630 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:56.812214 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.812925 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.814906 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.815735 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.815924 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.816009 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.816347 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.816475 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:19:56.816861 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.810159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42169 (local address 127.23.173.65:56031)
0927 22:19:56.811159 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.811173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.811328 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.811664 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.811668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.811690 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:56.812120 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.812130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.813095 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.813100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.814788 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.814799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.816000 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.816005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.816140 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.816203 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.816305 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.816443 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":71,"thread_start_us":548,"threads_started":1}
I20260927 22:19:56.817306 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.809942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:43915 (local address 127.23.173.65:40843)
0927 22:19:56.811087 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:56.811107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:56.811293 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:56.811646 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:56.811651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:56.811673 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:56.812052 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.812061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.813063 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.813071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.814750 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:56.814761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.815965 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.815971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:56.816130 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.816202 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 22:19:56.816307 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:19:56.816455 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":109,"thread_start_us":579,"threads_started":1}
I20260927 22:19:56.817806 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de48a77cc648628c7b1b55390e49e5" candidate_uuid: "ff313e511b894013a2b6a47aa442afa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13511b6d4c754365b6b84a081b41657a" is_pre_election: true
I20260927 22:19:56.818333 11378 raft_consensus.cc:2454] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff313e511b894013a2b6a47aa442afa9 in term 0.
I20260927 22:19:56.818370 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de48a77cc648628c7b1b55390e49e5" candidate_uuid: "ff313e511b894013a2b6a47aa442afa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33dff89a47d4b14b8cfdefde211c237" is_pre_election: true
I20260927 22:19:56.818842 11512 raft_consensus.cc:2454] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff313e511b894013a2b6a47aa442afa9 in term 0.
I20260927 22:19:56.819049 11179 leader_election.cc:302] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13511b6d4c754365b6b84a081b41657a, ff313e511b894013a2b6a47aa442afa9; no voters:
I20260927 22:19:56.819482 11581 raft_consensus.cc:2781] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:56.819628 11581 raft_consensus.cc:482] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:56.819736 11581 raft_consensus.cc:3037] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:56.821847 11581 raft_consensus.cc:504] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.822475 11581 leader_election.cc:288] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [CANDIDATE]: Term 1 election: Requested vote from peers a33dff89a47d4b14b8cfdefde211c237 (127.23.173.67:43915), 13511b6d4c754365b6b84a081b41657a (127.23.173.66:42169)
I20260927 22:19:56.822917 11512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de48a77cc648628c7b1b55390e49e5" candidate_uuid: "ff313e511b894013a2b6a47aa442afa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33dff89a47d4b14b8cfdefde211c237"
I20260927 22:19:56.822952 11378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3de48a77cc648628c7b1b55390e49e5" candidate_uuid: "ff313e511b894013a2b6a47aa442afa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13511b6d4c754365b6b84a081b41657a"
I20260927 22:19:56.823110 11512 raft_consensus.cc:3037] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:56.823122 11378 raft_consensus.cc:3037] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:56.825201 11378 raft_consensus.cc:2454] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff313e511b894013a2b6a47aa442afa9 in term 1.
I20260927 22:19:56.825234 11512 raft_consensus.cc:2454] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff313e511b894013a2b6a47aa442afa9 in term 1.
I20260927 22:19:56.825551 11179 leader_election.cc:302] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13511b6d4c754365b6b84a081b41657a, ff313e511b894013a2b6a47aa442afa9; no voters:
I20260927 22:19:56.825829 11581 raft_consensus.cc:2781] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:56.826277 11581 raft_consensus.cc:686] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 1 LEADER]: Becoming Leader. State: Replica: ff313e511b894013a2b6a47aa442afa9, State: Running, Role: LEADER
I20260927 22:19:56.826782 11581 consensus_queue.cc:237] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } }
I20260927 22:19:56.831542 11099 catalog_manager.cc:5495] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff313e511b894013a2b6a47aa442afa9 (127.23.173.65). New cstate: current_term: 1 leader_uuid: "ff313e511b894013a2b6a47aa442afa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff313e511b894013a2b6a47aa442afa9" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 37081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:56.950260 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.944200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.23.173.65:37081)
0927 22:19:56.944745 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.944753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.944978 (+ 225us) server_negotiation.cc:407] Connection header received
0927 22:19:56.945156 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.945160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.945256 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.945388 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:56.946033 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.946721 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.948152 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:56.948550 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:56.948646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:56.949145 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:56.949681 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:56.949797 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:56.949917 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:19:56.950050 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
I20260927 22:19:56.957942 11512 raft_consensus.cc:1261] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 1 FOLLOWER]: Refusing update from remote peer ff313e511b894013a2b6a47aa442afa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:56.958079 11378 raft_consensus.cc:1261] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 1 FOLLOWER]: Refusing update from remote peer ff313e511b894013a2b6a47aa442afa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:56.958945 11586 consensus_queue.cc:1035] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33dff89a47d4b14b8cfdefde211c237" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 43915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:56.959295 11581 consensus_queue.cc:1035] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13511b6d4c754365b6b84a081b41657a" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:56.982460 11100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:56.982718 11100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:56.986066 11100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dc21bc396ea4c8cacd5e94868acccfc: Sending DeleteTablet for 3 replicas of tablet c3de48a77cc648628c7b1b55390e49e5
I20260927 22:19:56.987109 11224 tablet_service.cc:1514] Processing DeleteTablet for tablet c3de48a77cc648628c7b1b55390e49e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:56 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:59302
I20260927 22:19:56.987142 11492 tablet_service.cc:1514] Processing DeleteTablet for tablet c3de48a77cc648628c7b1b55390e49e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:56 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:36882
I20260927 22:19:56.988021 11603 tablet_replica.cc:330] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: stopping tablet replica
I20260927 22:19:56.988538 11358 tablet_service.cc:1514] Processing DeleteTablet for tablet c3de48a77cc648628c7b1b55390e49e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:56 UTC) from {username='kudu', principal='kudu/127.23.173.126@KRBTEST.COM'} at 127.0.0.1:35216
I20260927 22:19:56.988564 11603 raft_consensus.cc:2229] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:56.989866 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11161
I20260927 22:19:56.989975 11604 tablet_replica.cc:330] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: stopping tablet replica
I20260927 22:19:56.990401 11604 raft_consensus.cc:2229] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:56.990844 11604 raft_consensus.cc:2258] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:56.993062 11604 ts_tablet_manager.cc:1905] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:56.993198 11603 raft_consensus.cc:2258] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:56.994972 11603 ts_tablet_manager.cc:1905] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:56.999468 11605 tablet_replica.cc:330] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: stopping tablet replica
I20260927 22:19:56.999857 11605 raft_consensus.cc:2229] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:57.000021 11604 ts_tablet_manager.cc:1918] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:57.000176 11604 log.cc:1196] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/c3de48a77cc648628c7b1b55390e49e5
I20260927 22:19:57.000428 11605 raft_consensus.cc:2258] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:57.000669 11604 ts_tablet_manager.cc:1939] T c3de48a77cc648628c7b1b55390e49e5 P a33dff89a47d4b14b8cfdefde211c237: Deleting consensus metadata
I20260927 22:19:57.002236 11605 ts_tablet_manager.cc:1905] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:57.003332 11084 catalog_manager.cc:4841] TS a33dff89a47d4b14b8cfdefde211c237 (127.23.173.67:43915): tablet c3de48a77cc648628c7b1b55390e49e5 (table test-table [id=7a02a8f7f840499ca09967cbc7e712fb]) successfully deleted
I20260927 22:19:57.004099 11603 ts_tablet_manager.cc:1918] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:57.004267 11603 log.cc:1196] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-0/wal/wals/c3de48a77cc648628c7b1b55390e49e5
I20260927 22:19:57.004768 11603 ts_tablet_manager.cc:1939] T c3de48a77cc648628c7b1b55390e49e5 P ff313e511b894013a2b6a47aa442afa9: Deleting consensus metadata
I20260927 22:19:57.006096 11085 catalog_manager.cc:4841] TS ff313e511b894013a2b6a47aa442afa9 (127.23.173.65:37081): tablet c3de48a77cc648628c7b1b55390e49e5 (table test-table [id=7a02a8f7f840499ca09967cbc7e712fb]) successfully deleted
Sep 27 22:19:57 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.65@KRBTEST.COM
I20260927 22:19:57.008550 11605 ts_tablet_manager.cc:1918] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:57.008710 11605 log.cc:1196] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/c3de48a77cc648628c7b1b55390e49e5
I20260927 22:19:57.009280 11605 ts_tablet_manager.cc:1939] T c3de48a77cc648628c7b1b55390e49e5 P 13511b6d4c754365b6b84a081b41657a: Deleting consensus metadata
I20260927 22:19:57.011112 11086 catalog_manager.cc:4841] TS 13511b6d4c754365b6b84a081b41657a (127.23.173.66:42169): tablet c3de48a77cc648628c7b1b55390e49e5 (table test-table [id=7a02a8f7f840499ca09967cbc7e712fb]) successfully deleted
I20260927 22:19:57.016050 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:56.991943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.23.173.65:37081)
0927 22:19:56.992101 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:56.992109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:56.992132 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:19:56.992184 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:56.992189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:56.992261 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:56.992365 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:56.998965 (+ 6600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.000149 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.002229 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.002924 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.003033 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:57.009055 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:57.009070 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:57.009090 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:57.009138 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:57.012874 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.013724 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.013730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.013733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.013850 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.014541 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.014544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.014546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.014885 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:57.015098 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:57.015541 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 22:19:57.015730 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:19:57.016782 11284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:57.112496 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11161
I20260927 22:19:57.131138 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11295
Sep 27 22:19:57 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.66@KRBTEST.COM
I20260927 22:19:57.146848 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:57.131901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.23.173.66:42169)
0927 22:19:57.132110 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:57.132117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:57.132230 (+ 113us) server_negotiation.cc:407] Connection header received
0927 22:19:57.132459 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:57.132466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:57.132536 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:57.132664 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:57.134025 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.134839 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.136258 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.136911 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.137033 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:57.141296 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:57.141312 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:57.141327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:57.141359 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:57.144114 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.144865 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.144872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.144875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.144992 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.145326 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.145330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.145332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.145704 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:57.145916 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:57.146428 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 22:19:57.146599 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:19:57.147549 11418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:57.240460 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11295
I20260927 22:19:57.258076 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11429
Sep 27 22:19:57 dist-test-slave-3wc3 krb5kdc[11055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.67@KRBTEST.COM
I20260927 22:19:57.272822 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:57.258794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.23.173.67:43915)
0927 22:19:57.258964 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:57.258971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:57.258989 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:19:57.259148 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:57.259153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:57.259215 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:57.259297 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:57.260577 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.261315 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.262694 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:57.263102 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:57.263192 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:57.267653 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:57.267666 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:57.267681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:19:57.267724 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:57.270415 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.271019 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.271027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.271029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.271158 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:19:57.271476 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:19:57.271479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:19:57.271482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:19:57.271824 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:57.272028 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:57.272293 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:19:57.272447 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:19:57.273480 11552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:57.360774 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11429
I20260927 22:19:57.375036 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11068
I20260927 22:19:57.376188 11130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:57.487440 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11068
2026-09-27T22:19:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:19:57Z Disabled control of system clock
I20260927 22:19:57.564796 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41021
--webserver_interface=127.23.173.126
--webserver_port=0
--builtin_ntp_servers=127.23.173.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:57.891691 11614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:57.891990 11614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:57.892167 11614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:57.899660 11614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:19:57.899767 11614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:57.899864 11614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:57.899969 11614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:19:57.900031 11614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:19:57.910815 11614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:41021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:57.912150 11614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:57.913913 11614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:57.926108 11619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:57.926028 11622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:57.926154 11620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:57.926960 11614 server_base.cc:1034] running on GCE node
I20260927 22:19:57.927814 11614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:57.929193 11614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:57.930457 11614 hybrid_clock.cc:648] HybridClock initialized: now 1790547597930392 us; error 56 us; skew 500 ppm
I20260927 22:19:57.930824 11614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:57.933216 11614 webserver.cc:460] Webserver started at http://127.23.173.126:34037/ using document root <none> and password file <none>
I20260927 22:19:57.933907 11614 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:57.934074 11614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:57.934427 11614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:57.937305 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "1c4b7e059aa6489ea4953565e10d1732"
format_stamp: "Formatted at 2026-09-27 22:19:57 on dist-test-slave-3wc3"
server_key: "575d3d6c799c8664f96bd216d375044d"
server_key_iv: "97a4232467791b08e5e37cad6592f536"
server_key_version: "encryptionkey@0"
I20260927 22:19:57.938203 11614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "1c4b7e059aa6489ea4953565e10d1732"
format_stamp: "Formatted at 2026-09-27 22:19:57 on dist-test-slave-3wc3"
server_key: "575d3d6c799c8664f96bd216d375044d"
server_key_iv: "97a4232467791b08e5e37cad6592f536"
server_key_version: "encryptionkey@0"
I20260927 22:19:57.943162 11614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:19:57.946691 11628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:57.948017 11614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:57.948211 11614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "1c4b7e059aa6489ea4953565e10d1732"
format_stamp: "Formatted at 2026-09-27 22:19:57 on dist-test-slave-3wc3"
server_key: "575d3d6c799c8664f96bd216d375044d"
server_key_iv: "97a4232467791b08e5e37cad6592f536"
server_key_version: "encryptionkey@0"
I20260927 22:19:57.948442 11614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:57.969900 11614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:57.970863 11614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:57.971189 11614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:57.990296 11614 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:41021
I20260927 22:19:57.990303 11679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:41021 every 8 connection(s)
I20260927 22:19:57.991675 11614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:19:57.995357 11680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:57.997349 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11614
I20260927 22:19:57.997584 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:19:57.998106 24245 external_mini_cluster.cc:1428] Setting key 7d77174653b6ac4ed341f83cf95f2e67
I20260927 22:19:58.006908 11680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: Bootstrap starting.
I20260927 22:19:58.010664 11680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:58.011761 11680 log.cc:824] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:58.012099 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:57.999973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.23.173.126:41021)
0927 22:19:58.000995 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:58.001005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:58.001047 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:19:58.002733 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:58.002755 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:58.003188 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:58.003542 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:58.004138 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.005545 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.007414 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.008077 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.008209 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:58.009571 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:58.009579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:58.009593 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:58.010225 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:58.010324 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:58.010348 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:58.010442 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:58.010710 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:19:58.011091 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":403,"threads_started":1}
I20260927 22:19:58.015188 11680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: No bootstrap required, opened a new log
I20260927 22:19:58.022464 11680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } }
I20260927 22:19:58.022876 11680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:58.023000 11680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4b7e059aa6489ea4953565e10d1732, State: Initialized, Role: FOLLOWER
I20260927 22:19:58.023582 11680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } }
I20260927 22:19:58.023835 11680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:19:58.023986 11680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:19:58.024164 11680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:58.026648 11680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } }
I20260927 22:19:58.027154 11680 leader_election.cc:302] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4b7e059aa6489ea4953565e10d1732; no voters:
I20260927 22:19:58.027750 11680 leader_election.cc:288] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:19:58.027947 11685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:58.028752 11685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [term 1 LEADER]: Becoming Leader. State: Replica: 1c4b7e059aa6489ea4953565e10d1732, State: Running, Role: LEADER
I20260927 22:19:58.029253 11685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } }
I20260927 22:19:58.029906 11680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:19:58.032140 11686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4b7e059aa6489ea4953565e10d1732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } } }
I20260927 22:19:58.032611 11686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:58.032313 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4b7e059aa6489ea4953565e10d1732. Latest consensus state: current_term: 1 leader_uuid: "1c4b7e059aa6489ea4953565e10d1732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4b7e059aa6489ea4953565e10d1732" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41021 } } }
I20260927 22:19:58.032882 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732 [sys.catalog]: This master's current role is: LEADER
I20260927 22:19:58.035073 11694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:19:58.040377 11694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:19:58.048971 11694 catalog_manager.cc:1347] Generated new cluster ID: 937a946f75b849b5a597730f8c943677
I20260927 22:19:58.049090 11694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:19:58.067130 11694 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:19:58.068441 11694 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:19:58.077443 11694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: Generated new TSK 0
I20260927 22:19:58.078344 11694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:19:58.085076 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.65:0
--local_ip_for_outbound_sockets=127.23.173.65
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--builtin_ntp_servers=127.23.173.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:19:58.440243 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:58.440590 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:58.440752 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:58.448778 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:58.448945 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:58.449142 11704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.65
I20260927 22:19:58.458891 11704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.173.65
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.65
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:58.460250 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:58.462108 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:58.475315 11709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:58.475229 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:58.475273 11710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:58.476220 11704 server_base.cc:1034] running on GCE node
I20260927 22:19:58.476847 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:58.477865 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:58.479131 11704 hybrid_clock.cc:648] HybridClock initialized: now 1790547598479086 us; error 39 us; skew 500 ppm
I20260927 22:19:58.479475 11704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:58.481763 11704 webserver.cc:460] Webserver started at http://127.23.173.65:40277/ using document root <none> and password file <none>
I20260927 22:19:58.482517 11704 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:58.482643 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:58.482970 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:58.485782 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data/instance:
uuid: "04468679682f47d5b445fdb6dee39a53"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "cd5d898b2ba6287f88dd2723c6ebcf05"
server_key_iv: "31b5c1037272f8812516e355f8d3ccf7"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.486692 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance:
uuid: "04468679682f47d5b445fdb6dee39a53"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "cd5d898b2ba6287f88dd2723c6ebcf05"
server_key_iv: "31b5c1037272f8812516e355f8d3ccf7"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.491811 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:19:58.495178 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.496541 11704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:19:58.496726 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal
uuid: "04468679682f47d5b445fdb6dee39a53"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "cd5d898b2ba6287f88dd2723c6ebcf05"
server_key_iv: "31b5c1037272f8812516e355f8d3ccf7"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.496965 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:58.516682 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:58.517490 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:58.518085 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:58.519114 11704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:58.520642 11704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:58.520797 11704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.520924 11704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:58.521014 11704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.554559 11704 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.65:33045
I20260927 22:19:58.554651 11830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.65:33045 every 8 connection(s)
I20260927 22:19:58.556069 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/data/info.pb
I20260927 22:19:58.557744 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11704
I20260927 22:19:58.557914 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-0/wal/instance
I20260927 22:19:58.558385 24245 external_mini_cluster.cc:1428] Setting key e777a3a1018c0255a2f70d09ecc1e52f
I20260927 22:19:58.561373 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.66:0
--local_ip_for_outbound_sockets=127.23.173.66
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--builtin_ntp_servers=127.23.173.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:58.569679 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:58.558730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.65:37075 (local address 127.23.173.126:41021)
0927 22:19:58.559506 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:58.559512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:58.560728 (+ 1216us) server_negotiation.cc:407] Connection header received
0927 22:19:58.562644 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:58.562652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:58.562742 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:58.562857 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:58.563941 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.565075 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.566760 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.567476 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.567614 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:58.568938 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:58.568947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:58.568955 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:58.568980 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:58.569035 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:58.569053 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:58.569150 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:58.569334 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:19:58.569420 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":511,"threads_started":1}
I20260927 22:19:58.570708 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:58.559358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41021 (local address 127.23.173.65:37075)
0927 22:19:58.560585 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:58.560627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:58.562377 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:58.563053 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:58.563061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:58.563146 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:58.563767 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:58.563784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.565219 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.565223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:58.566620 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:58.566632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:58.567687 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:58.567692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:58.567890 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:58.568622 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:58.568625 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:58.568663 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:58.568745 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:58.569151 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:58.569159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:58.569164 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:19:58.569305 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:19:58.569629 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":324,"threads_started":1}
I20260927 22:19:58.572278 11831 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41021
I20260927 22:19:58.572598 11831 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:58.573331 11831 heartbeater.cc:511] Master 127.23.173.126:41021 requested a full tablet report, sending...
I20260927 22:19:58.575327 11645 ts_manager.cc:194] Registered new tserver with Master: 04468679682f47d5b445fdb6dee39a53 (127.23.173.65:33045)
I20260927 22:19:58.577008 11645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.65:37075
W20260927 22:19:58.892074 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:58.892386 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:58.892553 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:58.899873 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:58.899987 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:58.900189 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.66
I20260927 22:19:58.909701 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.173.66
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.66
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:58.911191 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:58.913125 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:58.926168 11840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:58.926133 11841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:58.926193 11843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:58.927397 11835 server_base.cc:1034] running on GCE node
I20260927 22:19:58.927983 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:58.928949 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:58.930166 11835 hybrid_clock.cc:648] HybridClock initialized: now 1790547598930127 us; error 62 us; skew 500 ppm
I20260927 22:19:58.930470 11835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:58.932619 11835 webserver.cc:460] Webserver started at http://127.23.173.66:32781/ using document root <none> and password file <none>
I20260927 22:19:58.933282 11835 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:58.933410 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:58.933724 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:58.936553 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data/instance:
uuid: "12fb3abaddc24fb78e7734f47e3857e8"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "6f22c913b18b0c7e277c8c1ecd96d4d1"
server_key_iv: "01eebb571c5a31e64c337b56b62657e9"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.937317 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance:
uuid: "12fb3abaddc24fb78e7734f47e3857e8"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "6f22c913b18b0c7e277c8c1ecd96d4d1"
server_key_iv: "01eebb571c5a31e64c337b56b62657e9"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.942255 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:19:58.945501 11849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.946751 11835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:19:58.946940 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal
uuid: "12fb3abaddc24fb78e7734f47e3857e8"
format_stamp: "Formatted at 2026-09-27 22:19:58 on dist-test-slave-3wc3"
server_key: "6f22c913b18b0c7e277c8c1ecd96d4d1"
server_key_iv: "01eebb571c5a31e64c337b56b62657e9"
server_key_version: "encryptionkey@0"
I20260927 22:19:58.947153 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:58.960041 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:58.961112 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:58.961434 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:58.962379 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:58.963780 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:58.963864 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.963944 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:58.963999 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:58.996445 11835 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.66:42109
I20260927 22:19:58.996476 11961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.66:42109 every 8 connection(s)
I20260927 22:19:58.998031 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/data/info.pb
I20260927 22:19:58.998828 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11835
I20260927 22:19:58.999001 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal/instance
I20260927 22:19:58.999547 24245 external_mini_cluster.cc:1428] Setting key 4508e3399ba126540d56a634e7bcfefb
I20260927 22:19:59.003535 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.173.67:0
--local_ip_for_outbound_sockets=127.23.173.67
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--builtin_ntp_servers=127.23.173.84:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:19:59.024966 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.001456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.66:37611 (local address 127.23.173.126:41021)
0927 22:19:59.001642 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.001648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.015129 (+ 13481us) server_negotiation.cc:407] Connection header received
0927 22:19:59.017801 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.017807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.017901 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.018049 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:59.019153 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.020338 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.021903 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.022642 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.022778 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.024085 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:59.024093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:59.024101 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:59.024125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:59.024167 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:59.024184 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:59.024274 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.024475 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.024581 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:19:59.025815 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.001596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41021 (local address 127.23.173.66:37611)
0927 22:19:59.014934 (+ 13338us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.014980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.017484 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.018304 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.018313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.018405 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:59.018975 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.018992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.020530 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.020534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.021752 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.021765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.022836 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.022842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.023047 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.023772 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:59.023774 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:59.023812 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:59.023887 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:59.024277 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:59.024284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:59.024289 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.024434 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.024735 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12736,"thread_start_us":12496,"threads_started":1}
I20260927 22:19:59.027257 11962 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41021
I20260927 22:19:59.027594 11962 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:59.028455 11962 heartbeater.cc:511] Master 127.23.173.126:41021 requested a full tablet report, sending...
I20260927 22:19:59.029922 11645 ts_manager.cc:194] Registered new tserver with Master: 12fb3abaddc24fb78e7734f47e3857e8 (127.23.173.66:42109)
I20260927 22:19:59.030856 11645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.66:37611
W20260927 22:19:59.336511 11965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:19:59.336776 11965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:19:59.336944 11965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:19:59.344003 11965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:19:59.344090 11965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:19:59.344198 11965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.173.67
I20260927 22:19:59.353288 11965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.173.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.173.67
--webserver_port=0
--tserver_master_addrs=127.23.173.126:41021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.173.67
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:19:59.354557 11965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:19:59.356177 11965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:19:59.368492 11970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:59.368695 11971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:19:59.368762 11973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:19:59.369314 11965 server_base.cc:1034] running on GCE node
I20260927 22:19:59.369853 11965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:19:59.370774 11965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:19:59.371970 11965 hybrid_clock.cc:648] HybridClock initialized: now 1790547599371935 us; error 31 us; skew 500 ppm
I20260927 22:19:59.372236 11965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:19:59.374323 11965 webserver.cc:460] Webserver started at http://127.23.173.67:42581/ using document root <none> and password file <none>
I20260927 22:19:59.374926 11965 fs_manager.cc:359] Metadata directory not provided
I20260927 22:19:59.375010 11965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:19:59.375269 11965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:19:59.378093 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data/instance:
uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d"
format_stamp: "Formatted at 2026-09-27 22:19:59 on dist-test-slave-3wc3"
server_key: "47dc26b4fbf3a771254e512b5e492730"
server_key_iv: "6d634c0c7cc4fe5355b0b027be16ae65"
server_key_version: "encryptionkey@0"
I20260927 22:19:59.378801 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance:
uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d"
format_stamp: "Formatted at 2026-09-27 22:19:59 on dist-test-slave-3wc3"
server_key: "47dc26b4fbf3a771254e512b5e492730"
server_key_iv: "6d634c0c7cc4fe5355b0b027be16ae65"
server_key_version: "encryptionkey@0"
I20260927 22:19:59.384197 11965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 22:19:59.387367 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:59.388609 11965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:19:59.388733 11965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal
uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d"
format_stamp: "Formatted at 2026-09-27 22:19:59 on dist-test-slave-3wc3"
server_key: "47dc26b4fbf3a771254e512b5e492730"
server_key_iv: "6d634c0c7cc4fe5355b0b027be16ae65"
server_key_version: "encryptionkey@0"
I20260927 22:19:59.388895 11965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:19:59.405364 11965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:19:59.406172 11965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:19:59.406425 11965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:19:59.407291 11965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:19:59.408679 11965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:19:59.408761 11965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:59.408849 11965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:19:59.408902 11965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:19:59.441658 11965 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.67:40591
I20260927 22:19:59.441706 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.67:40591 every 8 connection(s)
I20260927 22:19:59.442975 11965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/data/info.pb
I20260927 22:19:59.448983 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 11965
I20260927 22:19:59.449189 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal/instance
I20260927 22:19:59.449676 24245 external_mini_cluster.cc:1428] Setting key 6df60c9ed1d98d5b0f647b0174630d1a
I20260927 22:19:59.455816 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.445619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:42021 (local address 127.23.173.126:41021)
0927 22:19:59.445808 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.445812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.447273 (+ 1461us) server_negotiation.cc:407] Connection header received
0927 22:19:59.449132 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.449140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.449228 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.449348 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:59.450765 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.451845 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.453461 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.453875 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.454034 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.455211 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:59.455216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:59.455219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:59.455236 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:59.455266 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:59.455277 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:59.455346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.455517 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.455590 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:19:59.456841 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.446241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:41021 (local address 127.23.173.67:42021)
0927 22:19:59.447122 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.447155 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.448783 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.449639 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.449648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.449737 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:19:59.450561 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.450581 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.451990 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.451994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.453346 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.453358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.454116 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.454121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.454319 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.454935 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:19:59.454937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:19:59.454974 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:19:59.455045 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:19:59.455341 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:19:59.455347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:19:59.455351 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.455473 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.455742 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":291,"threads_started":1}
I20260927 22:19:59.458196 12092 heartbeater.cc:347] Connected to a master server at 127.23.173.126:41021
I20260927 22:19:59.458501 12092 heartbeater.cc:464] Registering TS with master...
I20260927 22:19:59.459236 12092 heartbeater.cc:511] Master 127.23.173.126:41021 requested a full tablet report, sending...
I20260927 22:19:59.460536 11645 ts_manager.cc:194] Registered new tserver with Master: ea9f1fd45e744e4ca665a4107d5d4e6d (127.23.173.67:40591)
I20260927 22:19:59.461333 11645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.173.67:42021
I20260927 22:19:59.464893 24245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:19:59.474210 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.468763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.23.173.126:41021)
0927 22:19:59.468978 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.468987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.469333 (+ 346us) server_negotiation.cc:407] Connection header received
0927 22:19:59.469456 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.469459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.469538 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.469615 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:59.470066 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.470737 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.472211 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.472588 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.472672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.473563 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:59.473568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:59.473570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:59.473586 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:59.473615 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:59.473625 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:59.473691 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.473807 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.473881 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:19:59.477540 11645 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:19:59.484198 11645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:19:59.509076 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.495720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.23.173.65:33045)
0927 22:19:59.496452 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.496461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.497081 (+ 620us) server_negotiation.cc:407] Connection header received
0927 22:19:59.497297 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.497304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.497597 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.497810 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:59.499028 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.501272 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.504959 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.506953 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.507086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.507403 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.508304 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.508574 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":443,"threads_started":1}
I20260927 22:19:59.510150 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.495697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54212)
0927 22:19:59.496952 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.496979 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.497115 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.498180 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.498191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.498235 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:59.498761 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.498777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.501557 (+ 2780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.501566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.504676 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.504696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.507631 (+ 2935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.507640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.507980 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.508050 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.509391 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.509665 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"thread_start_us":1120,"threads_started":1}
I20260927 22:19:59.514483 11766 tablet_service.cc:1467] Processing CreateTablet for tablet cb9321dce4944494b0038b223ce1b62f (DEFAULT_TABLE table=test-table [id=b29d1a3dcccd4e0b914997d0096c7495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:59.515971 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9321dce4944494b0038b223ce1b62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:59.517047 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.495699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.67:40591 (local address 127.0.0.1:58540)
0927 22:19:59.508083 (+ 12384us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.508107 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.508557 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.510023 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.510031 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.510067 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:59.510774 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.510792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.512746 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.512749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.514762 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.514774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.516207 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.516214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.516461 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.516484 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.516590 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.516711 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12195,"mutex_wait_us":840,"thread_start_us":11745,"threads_started":1}
I20260927 22:19:59.517206 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.496198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.23.173.67:40591)
0927 22:19:59.496926 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.496935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.508224 (+ 11289us) server_negotiation.cc:407] Connection header received
0927 22:19:59.508811 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.508824 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.509140 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.509335 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:59.511047 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.512612 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.514943 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.515913 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.516079 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.516349 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.516644 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.516810 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"spinlock_wait_cycles":19840,"thread_start_us":496,"threads_started":1}
I20260927 22:19:59.519445 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.497590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42109 (local address 127.0.0.1:58764)
0927 22:19:59.510430 (+ 12840us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.510453 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.510589 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.512498 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.512502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.512528 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:59.512704 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.512714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.514597 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.514603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.516502 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.516533 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.518672 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.518678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.518864 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.518899 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.518993 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.519146 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12702,"mutex_wait_us":51,"thread_start_us":13266,"threads_started":1}
I20260927 22:19:59.519586 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.497948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.23.173.66:42109)
0927 22:19:59.511647 (+ 13699us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.511655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.511680 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:19:59.511777 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.511783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.512027 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.512220 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:59.512865 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.514437 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.516706 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.518202 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.518323 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.518580 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.519072 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.519244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13508,"thread_start_us":13374,"threads_started":1}
I20260927 22:19:59.520798 12027 tablet_service.cc:1467] Processing CreateTablet for tablet cb9321dce4944494b0038b223ce1b62f (DEFAULT_TABLE table=test-table [id=b29d1a3dcccd4e0b914997d0096c7495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:59.522063 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9321dce4944494b0038b223ce1b62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:59.522980 11897 tablet_service.cc:1467] Processing CreateTablet for tablet cb9321dce4944494b0038b223ce1b62f (DEFAULT_TABLE table=test-table [id=b29d1a3dcccd4e0b914997d0096c7495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:19:59.524308 11897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9321dce4944494b0038b223ce1b62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:19:59.530366 12110 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Bootstrap starting.
I20260927 22:19:59.534183 12110 tablet_bootstrap.cc:654] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:59.535794 12110 log.cc:824] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:59.537012 12112 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Bootstrap starting.
I20260927 22:19:59.538465 12111 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Bootstrap starting.
I20260927 22:19:59.540521 12112 tablet_bootstrap.cc:654] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:59.541733 12112 log.cc:824] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:59.541805 12111 tablet_bootstrap.cc:654] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Neither blocks nor log segments found. Creating new log.
I20260927 22:19:59.542845 12111 log.cc:824] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:19:59.543161 12110 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: No bootstrap required, opened a new log
I20260927 22:19:59.543503 12110 ts_tablet_manager.cc:1397] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260927 22:19:59.545401 12112 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: No bootstrap required, opened a new log
I20260927 22:19:59.545670 12112 ts_tablet_manager.cc:1397] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:19:59.545836 12111 tablet_bootstrap.cc:492] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: No bootstrap required, opened a new log
I20260927 22:19:59.546139 12111 ts_tablet_manager.cc:1397] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:19:59.550907 12110 raft_consensus.cc:348] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.551388 12110 raft_consensus.cc:374] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:59.551483 12110 raft_consensus.cc:729] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04468679682f47d5b445fdb6dee39a53, State: Initialized, Role: FOLLOWER
I20260927 22:19:59.552093 12110 consensus_queue.cc:260] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.553568 11831 heartbeater.cc:503] Master 127.23.173.126:41021 was elected leader, sending a full tablet report...
I20260927 22:19:59.555665 12110 ts_tablet_manager.cc:1428] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260927 22:19:59.558317 11832 tablet.cc:2369] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:59.558353 12112 raft_consensus.cc:348] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.558959 12112 raft_consensus.cc:374] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:59.559099 12112 raft_consensus.cc:729] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fb3abaddc24fb78e7734f47e3857e8, State: Initialized, Role: FOLLOWER
I20260927 22:19:59.559139 12111 raft_consensus.cc:348] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.559548 12111 raft_consensus.cc:374] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:19:59.559656 12111 raft_consensus.cc:729] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9f1fd45e744e4ca665a4107d5d4e6d, State: Initialized, Role: FOLLOWER
I20260927 22:19:59.560031 12112 consensus_queue.cc:260] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.560374 12111 consensus_queue.cc:260] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.562619 11962 heartbeater.cc:503] Master 127.23.173.126:41021 was elected leader, sending a full tablet report...
I20260927 22:19:59.563205 12092 heartbeater.cc:503] Master 127.23.173.126:41021 was elected leader, sending a full tablet report...
I20260927 22:19:59.563313 12112 ts_tablet_manager.cc:1428] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260927 22:19:59.564908 12111 ts_tablet_manager.cc:1428] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Time spent starting tablet: real 0.019s user 0.012s sys 0.003s
W20260927 22:19:59.694644 12093 tablet.cc:2369] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:19:59.765550 11963 tablet.cc:2369] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:19:59.792965 12118 raft_consensus.cc:482] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:19:59.793344 12118 raft_consensus.cc:504] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.795590 12118 leader_election.cc:288] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04468679682f47d5b445fdb6dee39a53 (127.23.173.65:33045), 12fb3abaddc24fb78e7734f47e3857e8 (127.23.173.66:42109)
I20260927 22:19:59.804061 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.795779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.23.173.67:56671)
0927 22:19:59.795971 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.795991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.796118 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.796831 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.796835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.796867 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:59.797032 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.797039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.798608 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.798613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.800680 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.800694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.802912 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.802921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.803166 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.803209 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.803317 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.803539 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260927 22:19:59.804109 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.796481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:42919 (local address 127.23.173.66:42109)
0927 22:19:59.796690 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.796697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.797123 (+ 426us) server_negotiation.cc:407] Connection header received
0927 22:19:59.797319 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.797324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.797396 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.797479 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:59.798014 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.799101 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.801543 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.802938 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.803109 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.803201 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.803735 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.803868 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:19:59.804519 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.796199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.66:42109 (local address 127.23.173.67:42919)
0927 22:19:59.797019 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 22:19:59.797032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:19:59.797178 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:19:59.797509 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:19:59.797514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:19:59.797538 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:19:59.797846 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.797854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.799276 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.799281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.801340 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:19:59.801356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.803308 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.803314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:19:59.803554 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.803596 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:19:59.803686 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:19:59.803807 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":24,"thread_start_us":490,"threads_started":1}
I20260927 22:19:59.804957 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.795873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.173.67:56671 (local address 127.23.173.65:33045)
0927 22:19:59.796078 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.796084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.796103 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:19:59.796240 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.796245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.796350 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.796437 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:19:59.797191 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.798459 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.800997 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.802515 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.802698 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.802786 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.804349 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.804511 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:19:59.805536 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9321dce4944494b0038b223ce1b62f" candidate_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fb3abaddc24fb78e7734f47e3857e8" is_pre_election: true
I20260927 22:19:59.806124 11917 raft_consensus.cc:2454] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9f1fd45e744e4ca665a4107d5d4e6d in term 0.
I20260927 22:19:59.806610 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9321dce4944494b0038b223ce1b62f" candidate_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04468679682f47d5b445fdb6dee39a53" is_pre_election: true
I20260927 22:19:59.806802 11983 leader_election.cc:302] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fb3abaddc24fb78e7734f47e3857e8, ea9f1fd45e744e4ca665a4107d5d4e6d; no voters:
I20260927 22:19:59.807152 11786 raft_consensus.cc:2454] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea9f1fd45e744e4ca665a4107d5d4e6d in term 0.
I20260927 22:19:59.807232 12118 raft_consensus.cc:2781] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:19:59.807399 12118 raft_consensus.cc:482] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:19:59.807497 12118 raft_consensus.cc:3037] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:59.810508 12118 raft_consensus.cc:504] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.811196 12118 leader_election.cc:288] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [CANDIDATE]: Term 1 election: Requested vote from peers 04468679682f47d5b445fdb6dee39a53 (127.23.173.65:33045), 12fb3abaddc24fb78e7734f47e3857e8 (127.23.173.66:42109)
I20260927 22:19:59.811735 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9321dce4944494b0038b223ce1b62f" candidate_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fb3abaddc24fb78e7734f47e3857e8"
I20260927 22:19:59.811791 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9321dce4944494b0038b223ce1b62f" candidate_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04468679682f47d5b445fdb6dee39a53"
I20260927 22:19:59.811966 11917 raft_consensus.cc:3037] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:59.811975 11786 raft_consensus.cc:3037] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:19:59.815213 11786 raft_consensus.cc:2454] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9f1fd45e744e4ca665a4107d5d4e6d in term 1.
I20260927 22:19:59.815213 11917 raft_consensus.cc:2454] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea9f1fd45e744e4ca665a4107d5d4e6d in term 1.
I20260927 22:19:59.816226 11983 leader_election.cc:302] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04468679682f47d5b445fdb6dee39a53, ea9f1fd45e744e4ca665a4107d5d4e6d; no voters:
I20260927 22:19:59.816596 12118 raft_consensus.cc:2781] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:19:59.817063 12118 raft_consensus.cc:686] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 1 LEADER]: Becoming Leader. State: Replica: ea9f1fd45e744e4ca665a4107d5d4e6d, State: Running, Role: LEADER
I20260927 22:19:59.817667 12118 consensus_queue.cc:237] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } }
I20260927 22:19:59.823014 11644 catalog_manager.cc:5495] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d reported cstate change: term changed from 0 to 1, leader changed from <none> to ea9f1fd45e744e4ca665a4107d5d4e6d (127.23.173.67). New cstate: current_term: 1 leader_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea9f1fd45e744e4ca665a4107d5d4e6d" member_type: VOTER last_known_addr { host: "127.23.173.67" port: 40591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 22:19:59.871209 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.865629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.23.173.67:40591)
0927 22:19:59.865842 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.865851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.865870 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:19:59.866140 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.866144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.866232 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.866383 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:19:59.866754 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.867522 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.869003 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.869425 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.869563 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.870093 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:19:59.870606 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:19:59.870770 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.870834 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.870957 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:19:59.879738 11786 raft_consensus.cc:1261] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 1 FOLLOWER]: Refusing update from remote peer ea9f1fd45e744e4ca665a4107d5d4e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:59.879734 11917 raft_consensus.cc:1261] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 1 FOLLOWER]: Refusing update from remote peer ea9f1fd45e744e4ca665a4107d5d4e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:19:59.880728 12120 consensus_queue.cc:1035] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12fb3abaddc24fb78e7734f47e3857e8" member_type: VOTER last_known_addr { host: "127.23.173.66" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:59.881140 12118 consensus_queue.cc:1035] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04468679682f47d5b445fdb6dee39a53" member_type: VOTER last_known_addr { host: "127.23.173.65" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:19:59.906906 11644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:59.907226 11644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:19:59.910923 11644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c4b7e059aa6489ea4953565e10d1732: Sending DeleteTablet for 3 replicas of tablet cb9321dce4944494b0038b223ce1b62f
I20260927 22:19:59.912048 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9321dce4944494b0038b223ce1b62f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:59 UTC) from {username='slave'} at 127.0.0.1:58540
I20260927 22:19:59.912118 11897 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9321dce4944494b0038b223ce1b62f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:59 UTC) from {username='slave'} at 127.0.0.1:58764
I20260927 22:19:59.912065 11766 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9321dce4944494b0038b223ce1b62f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:19:59 UTC) from {username='slave'} at 127.0.0.1:54212
I20260927 22:19:59.920625 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11704
I20260927 22:19:59.926786 12134 tablet_replica.cc:330] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: stopping tablet replica
I20260927 22:19:59.927336 12134 raft_consensus.cc:2229] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:59.927968 12134 raft_consensus.cc:2258] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:59.928894 12133 tablet_replica.cc:330] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: stopping tablet replica
I20260927 22:19:59.929371 12133 raft_consensus.cc:2229] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:19:59.930621 12133 raft_consensus.cc:2258] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:59.932636 12134 ts_tablet_manager.cc:1905] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:59.937116 12133 ts_tablet_manager.cc:1905] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:59.938500 12135 tablet_replica.cc:330] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: stopping tablet replica
I20260927 22:19:59.938910 12135 raft_consensus.cc:2229] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:19:59.939421 12135 raft_consensus.cc:2258] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:19:59.942221 12135 ts_tablet_manager.cc:1905] T cb9321dce4944494b0038b223ce1b62f P 04468679682f47d5b445fdb6dee39a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:19:59.943502 12134 ts_tablet_manager.cc:1918] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:59.943681 12134 log.cc:1196] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-1/wal/wals/cb9321dce4944494b0038b223ce1b62f
I20260927 22:19:59.944231 12134 ts_tablet_manager.cc:1939] T cb9321dce4944494b0038b223ce1b62f P 12fb3abaddc24fb78e7734f47e3857e8: Deleting consensus metadata
I20260927 22:19:59.944231 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:19:59.922660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.23.173.65:33045)
0927 22:19:59.922903 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:19:59.922913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:19:59.933726 (+ 10813us) server_negotiation.cc:407] Connection header received
0927 22:19:59.934244 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:19:59.934250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:19:59.934388 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:19:59.934466 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:19:59.935229 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.936452 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.938461 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:19:59.939171 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:19:59.939265 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:19:59.942468 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:19:59.942478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:19:59.942493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:19:59.943217 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:19:59.943287 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:19:59.943311 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:19:59.943419 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:19:59.943749 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 22:19:59.943871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:19:59.945580 11826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:19:59.946892 12133 ts_tablet_manager.cc:1918] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:19:59.947080 12133 log.cc:1196] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Deleting WAL directory at /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790547283249637-24245-0/minicluster-data/ts-2/wal/wals/cb9321dce4944494b0038b223ce1b62f
I20260927 22:19:59.947021 11632 catalog_manager.cc:4841] TS 12fb3abaddc24fb78e7734f47e3857e8 (127.23.173.66:42109): tablet cb9321dce4944494b0038b223ce1b62f (table test-table [id=b29d1a3dcccd4e0b914997d0096c7495]) successfully deleted
I20260927 22:19:59.947701 12133 ts_tablet_manager.cc:1939] T cb9321dce4944494b0038b223ce1b62f P ea9f1fd45e744e4ca665a4107d5d4e6d: Deleting consensus metadata
I20260927 22:19:59.949122 11629 catalog_manager.cc:4841] TS ea9f1fd45e744e4ca665a4107d5d4e6d (127.23.173.67:40591): tablet cb9321dce4944494b0038b223ce1b62f (table test-table [id=b29d1a3dcccd4e0b914997d0096c7495]) successfully deleted
I20260927 22:20:00.050701 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11704
W20260927 22:20:00.071885 11632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.173.65:33045 (error 108)
I20260927 22:20:00.073230 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11835
I20260927 22:20:00.074079 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.073188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54234)
0927 22:20:00.073680 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.073840 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":277,"threads_started":1}
W20260927 22:20:00.074414 11632 catalog_manager.cc:4584] TS 04468679682f47d5b445fdb6dee39a53 (127.23.173.65:33045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb9321dce4944494b0038b223ce1b62f: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
I20260927 22:20:00.081049 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.074161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.23.173.66:42109)
0927 22:20:00.074349 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:00.074355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:00.074377 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:20:00.074442 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:00.074447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:00.074558 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:00.074672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:00.075109 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:00.076214 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:00.077461 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:00.078227 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:00.078357 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:00.079422 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:00.079433 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:00.079453 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:00.080235 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:00.080297 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:00.080322 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:00.080441 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:00.080582 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:20:00.080689 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:20:00.081666 11957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:00.128093 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.127549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54240)
0927 22:20:00.127791 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.127914 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:20:00.129357 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.128885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54242)
0927 22:20:00.129078 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.129182 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:20:00.172420 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11835
I20260927 22:20:00.190482 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.189790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54248)
0927 22:20:00.190144 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.190238 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260927 22:20:00.190697 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11965
I20260927 22:20:00.191965 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.191260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54254)
0927 22:20:00.191693 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.191777 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
I20260927 22:20:00.199090 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.191600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.23.173.67:40591)
0927 22:20:00.191801 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:00.191807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:00.191827 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:20:00.191895 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:00.191900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:00.192001 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:00.192116 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:00.192624 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:00.194098 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:00.195381 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:00.196110 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:00.196244 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:00.197277 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:00.197290 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:00.197312 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:00.198268 (+ 956us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:00.198329 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:00.198354 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:00.198475 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:00.198609 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 22:20:00.198706 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:20:00.199906 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:00.272719 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.272198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54262)
0927 22:20:00.272459 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.272556 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:20:00.273888 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.273460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54276)
0927 22:20:00.273680 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.273753 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 22:20:00.322862 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11965
I20260927 22:20:00.342835 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 11614
I20260927 22:20:00.344117 11675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:00.451786 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:00.451073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.65:33045 (local address 127.0.0.1:54284)
0927 22:20:00.451394 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:00.451543 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.65:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260927 22:20:00.451783 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 11614
2026-09-27T22:20:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2964 ms)
[----------] 6 tests from EncryptionPolicyTest (22165 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:00 dist-test-slave-3wc3 krb5kdc[12145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:00 dist-test-slave-3wc3 krb5kdc[12145](info): set up 2 sockets
Sep 27 22:20:00 dist-test-slave-3wc3 krb5kdc[12145](info): commencing operation
krb5kdc: starting...
W20260927 22:20:01.554546 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:01 dist-test-slave-3wc3 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:01.814234 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:02.148411 12158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:02.148723 12158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:02.148907 12158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:02.156328 12158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:02.156445 12158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:02.156528 12158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:02.156627 12158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:02.156692 12158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:02.166836 12158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:02.168212 12158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:02.169473 12158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:02.181497 12164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:02.181459 12163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:02.181572 12166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:02.182641 12158 server_base.cc:1034] running on GCE node
I20260927 22:20:02.183493 12158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:02.184872 12158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:02.186076 12158 hybrid_clock.cc:648] HybridClock initialized: now 1790547602186012 us; error 63 us; skew 500 ppm
I20260927 22:20:02.186436 12158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:02 dist-test-slave-3wc3 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:02.189599 12158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:02.191212 12158 webserver.cc:460] Webserver started at http://127.0.0.1:38085/ using document root <none> and password file <none>
I20260927 22:20:02.191901 12158 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:02.192070 12158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:02.192374 12158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:02.195541 12158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "113720ac7bb5488682b491116d24c8ca"
format_stamp: "Formatted at 2026-09-27 22:20:02 on dist-test-slave-3wc3"
server_key: "2d5bc346b4aa79510c13b027fac57e46"
server_key_iv: "46dc8c3c0fac4cbfc22bf9a3b0252bcc"
server_key_version: "encryptionkey@0"
I20260927 22:20:02.196314 12158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "113720ac7bb5488682b491116d24c8ca"
format_stamp: "Formatted at 2026-09-27 22:20:02 on dist-test-slave-3wc3"
server_key: "2d5bc346b4aa79510c13b027fac57e46"
server_key_iv: "46dc8c3c0fac4cbfc22bf9a3b0252bcc"
server_key_version: "encryptionkey@0"
I20260927 22:20:02.201617 12158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:20:02.205119 12173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:02.206544 12158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:20:02.206734 12158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "113720ac7bb5488682b491116d24c8ca"
format_stamp: "Formatted at 2026-09-27 22:20:02 on dist-test-slave-3wc3"
server_key: "2d5bc346b4aa79510c13b027fac57e46"
server_key_iv: "46dc8c3c0fac4cbfc22bf9a3b0252bcc"
server_key_version: "encryptionkey@0"
I20260927 22:20:02.206951 12158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:02.239161 12158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:02.243853 12158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:02.244196 12158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:02.264187 12158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36819
I20260927 22:20:02.264210 12224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36819 every 8 connection(s)
I20260927 22:20:02.265669 12158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:02.269268 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12158
I20260927 22:20:02.269527 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:02.269758 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:02.270396 24245 external_mini_cluster.cc:1428] Setting key 0771e96c9e80537b26399a0dd0ef546c
I20260927 22:20:02.283861 12225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca: Bootstrap starting.
Sep 27 22:20:02 dist-test-slave-3wc3 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:20:02.287703 12225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:02.289115 12225 log.cc:824] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:02.292543 12225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca: No bootstrap required, opened a new log
I20260927 22:20:02.296177 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:02.272414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33223 (local address 127.0.0.1:36819)
0927 22:20:02.273352 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:02.273364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:02.273408 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:20:02.275945 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:02.275989 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:02.276425 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:02.276781 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:02.278551 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:02.279891 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:02.281689 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:02.282409 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:02.282597 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:02.287375 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:02.287387 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:02.287407 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:02.287466 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:02.291376 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:02.292198 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:02.292207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:02.292214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:02.292419 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:02.292908 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:02.292915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:02.292921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:02.293655 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:02.293995 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:02.294456 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 22:20:02.294808 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":453,"threads_started":1}
I20260927 22:20:02.300320 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:02.302503 12225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260927 22:20:02.302958 12225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:02.303081 12225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113720ac7bb5488682b491116d24c8ca, State: Initialized, Role: FOLLOWER
I20260927 22:20:02.303773 12225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260927 22:20:02.304131 12225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:02.304273 12225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:02.304469 12225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:02.307548 12225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260927 22:20:02.308110 12225 leader_election.cc:302] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 113720ac7bb5488682b491116d24c8ca; no voters:
I20260927 22:20:02.308771 12225 leader_election.cc:288] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:02.308884 12235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:02.310814 12235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [term 1 LEADER]: Becoming Leader. State: Replica: 113720ac7bb5488682b491116d24c8ca, State: Running, Role: LEADER
I20260927 22:20:02.311473 12225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:02.311398 12235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260927 22:20:02.319645 12236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "113720ac7bb5488682b491116d24c8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260927 22:20:02.320111 12236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:02.320773 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 113720ac7bb5488682b491116d24c8ca. Latest consensus state: current_term: 1 leader_uuid: "113720ac7bb5488682b491116d24c8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113720ac7bb5488682b491116d24c8ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260927 22:20:02.321118 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:02.329481 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:02.306396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50137 (local address 127.0.0.1:36819)
0927 22:20:02.306599 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:02.306607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:02.307150 (+ 543us) server_negotiation.cc:407] Connection header received
0927 22:20:02.307294 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:02.307302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:02.307389 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:02.307523 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:02.309182 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:02.310385 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:02.312266 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:02.312924 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:02.313096 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:02.315927 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:02.315938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:02.315941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:02.315975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:02.319608 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:02.320698 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:02.320704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:02.320708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:02.320888 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:02.321683 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:02.321691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:02.321702 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:02.326454 (+ 4752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:02.326840 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:02.328658 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0927 22:20:02.328935 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:20:02.333063 12243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:02.338315 12243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:02.347009 12243 catalog_manager.cc:1347] Generated new cluster ID: 36b509603fbf4a689b38cc08784288f4
I20260927 22:20:02.347131 12243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:02.362017 12243 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:02.363303 12243 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:02.372220 12243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 113720ac7bb5488682b491116d24c8ca: Generated new TSK 0
I20260927 22:20:02.373096 12243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:02.383754 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12158
I20260927 22:20:02.385469 12220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:02.476995 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12158
W20260927 22:20:02.495723 12143 connection.cc:535] client connection to 127.0.0.1:36819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36819): Connection reset by peer (error 104)
2026-09-27T22:20:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:02 dist-test-slave-3wc3 krb5kdc[12258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:02 dist-test-slave-3wc3 krb5kdc[12258](info): set up 2 sockets
Sep 27 22:20:02 dist-test-slave-3wc3 krb5kdc[12258](info): commencing operation
krb5kdc: starting...
W20260927 22:20:03.582389 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:03 dist-test-slave-3wc3 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:03.833823 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:04.172685 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:04.173036 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:04.173207 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:04.180845 12271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:04.180997 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:04.181105 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:04.181217 12271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:04.181293 12271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:04.191161 12271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:04.192464 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:04.193773 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:04.205483 12277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:04.205536 12276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:04.206182 12279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:04.206372 12271 server_base.cc:1034] running on GCE node
I20260927 22:20:04.207118 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:04.208562 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:04.209790 12271 hybrid_clock.cc:648] HybridClock initialized: now 1790547604209747 us; error 30 us; skew 500 ppm
I20260927 22:20:04.210198 12271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:04 dist-test-slave-3wc3 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:04.213220 12271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:04.214681 12271 webserver.cc:460] Webserver started at http://127.0.0.1:41233/ using document root <none> and password file <none>
I20260927 22:20:04.215368 12271 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:04.215515 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:04.215813 12271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:04.218744 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd"
format_stamp: "Formatted at 2026-09-27 22:20:04 on dist-test-slave-3wc3"
server_key: "688011b9d02a810759e8e4ae09e2e027"
server_key_iv: "a22eaf97708d94ae4c624768879e4e9e"
server_key_version: "encryptionkey@0"
I20260927 22:20:04.219511 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd"
format_stamp: "Formatted at 2026-09-27 22:20:04 on dist-test-slave-3wc3"
server_key: "688011b9d02a810759e8e4ae09e2e027"
server_key_iv: "a22eaf97708d94ae4c624768879e4e9e"
server_key_version: "encryptionkey@0"
I20260927 22:20:04.224556 12271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:20:04.227891 12286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:04.229230 12271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:04.229406 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd"
format_stamp: "Formatted at 2026-09-27 22:20:04 on dist-test-slave-3wc3"
server_key: "688011b9d02a810759e8e4ae09e2e027"
server_key_iv: "a22eaf97708d94ae4c624768879e4e9e"
server_key_version: "encryptionkey@0"
I20260927 22:20:04.229597 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:04.249758 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:04.253715 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:04.254050 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:04.273000 12271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35297
I20260927 22:20:04.273022 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35297 every 8 connection(s)
I20260927 22:20:04.274431 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:04.278102 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:04.278679 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12271
I20260927 22:20:04.278878 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:04.279330 24245 external_mini_cluster.cc:1428] Setting key 42aa3b93fa00ab2d73c2ce8423c8ca0d
I20260927 22:20:04.291682 12338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd: Bootstrap starting.
I20260927 22:20:04.293414 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:04.281104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59163 (local address 127.0.0.1:35297)
0927 22:20:04.281887 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:04.281899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:04.281939 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:20:04.284018 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:04.284043 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:04.284441 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:04.284769 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:04.285429 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:04.286835 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:04.288785 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:04.289546 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:04.289684 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:04.291026 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:04.291034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:04.291053 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:04.291644 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:04.291730 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:04.291754 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:04.291949 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:04.292264 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:20:04.292435 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":356,"threads_started":1}
I20260927 22:20:04.294965 12338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:04.296090 12338 log.cc:824] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:04.297313 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:04.298990 12338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd: No bootstrap required, opened a new log
I20260927 22:20:04.307070 12338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260927 22:20:04.307469 12338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:04.307646 12338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfdb4cdf48e46b0bfa18819322e0dfd, State: Initialized, Role: FOLLOWER
I20260927 22:20:04.307904 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:04.301697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34835 (local address 127.0.0.1:35297)
0927 22:20:04.302089 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:04.302095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:04.302328 (+ 233us) server_negotiation.cc:407] Connection header received
0927 22:20:04.302541 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:04.302545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:04.302602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:04.302713 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:04.303209 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:04.303999 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:04.305328 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:04.305800 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:04.305953 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:04.307060 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:04.307065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:04.307068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:04.307090 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:04.307139 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:04.307154 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:04.307274 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:04.307467 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 22:20:04.307519 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:20:04.308382 12338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260927 22:20:04.308699 12338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:04.308847 12338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:04.309032 12338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:04.311362 12338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260927 22:20:04.311793 12338 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfdb4cdf48e46b0bfa18819322e0dfd; no voters:
I20260927 22:20:04.312306 12338 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:04.312413 12348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:04.313200 12348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [term 1 LEADER]: Becoming Leader. State: Replica: 8cfdb4cdf48e46b0bfa18819322e0dfd, State: Running, Role: LEADER
I20260927 22:20:04.313827 12348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260927 22:20:04.314814 12338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:04.316856 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260927 22:20:04.317309 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:04.316829 12349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfdb4cdf48e46b0bfa18819322e0dfd. Latest consensus state: current_term: 1 leader_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfdb4cdf48e46b0bfa18819322e0dfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260927 22:20:04.318293 12349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:04.318949 12353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:04.324193 12353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:04.332664 12353 catalog_manager.cc:1347] Generated new cluster ID: 8ebf534c5af04567b8f349445c8cd82f
I20260927 22:20:04.332782 12353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:04.350219 12353 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:04.351502 12353 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:04.359753 12353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfdb4cdf48e46b0bfa18819322e0dfd: Generated new TSK 0
I20260927 22:20:04.360544 12353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:04.369393 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12271
W20260927 22:20:04.370736 12333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59163
W20260927 22:20:04.371538 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:04.371730 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12271
W20260927 22:20:04.389050 12256 connection.cc:535] client connection to 127.0.0.1:35297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35297): Connection reset by peer (error 104)
2026-09-27T22:20:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:04 dist-test-slave-3wc3 krb5kdc[12369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:04 dist-test-slave-3wc3 krb5kdc[12369](info): set up 2 sockets
Sep 27 22:20:04 dist-test-slave-3wc3 krb5kdc[12369](info): commencing operation
krb5kdc: starting...
W20260927 22:20:05.478336 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:05 dist-test-slave-3wc3 krb5kdc[12369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:05.737908 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:06.070134 12382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:06.070410 12382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:06.070595 12382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:06.078398 12382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:06.078500 12382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:06.078604 12382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:06.078688 12382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:06.078764 12382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:06.088493 12382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:06.089726 12382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:06.091032 12382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:06.102658 12387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:06.102952 12388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:06.103353 12390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:06.103688 12382 server_base.cc:1034] running on GCE node
I20260927 22:20:06.104451 12382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:06.105816 12382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:06.107017 12382 hybrid_clock.cc:648] HybridClock initialized: now 1790547606106976 us; error 60 us; skew 500 ppm
I20260927 22:20:06.107390 12382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:06 dist-test-slave-3wc3 krb5kdc[12369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:06.116580 12382 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:06.118039 12382 webserver.cc:460] Webserver started at http://127.0.0.1:34825/ using document root <none> and password file <none>
I20260927 22:20:06.118710 12382 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:06.118857 12382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:06.119163 12382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:06.122160 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "da43cec314144605a71238a110922710"
format_stamp: "Formatted at 2026-09-27 22:20:06 on dist-test-slave-3wc3"
server_key: "8b6240ea6fa85a57ed697209edb7cc8d"
server_key_iv: "d613ae1c1071d8df960574024080435f"
server_key_version: "encryptionkey@0"
I20260927 22:20:06.122923 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "da43cec314144605a71238a110922710"
format_stamp: "Formatted at 2026-09-27 22:20:06 on dist-test-slave-3wc3"
server_key: "8b6240ea6fa85a57ed697209edb7cc8d"
server_key_iv: "d613ae1c1071d8df960574024080435f"
server_key_version: "encryptionkey@0"
I20260927 22:20:06.127909 12382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:20:06.131453 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:06.132776 12382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:06.132957 12382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "da43cec314144605a71238a110922710"
format_stamp: "Formatted at 2026-09-27 22:20:06 on dist-test-slave-3wc3"
server_key: "8b6240ea6fa85a57ed697209edb7cc8d"
server_key_iv: "d613ae1c1071d8df960574024080435f"
server_key_version: "encryptionkey@0"
I20260927 22:20:06.133163 12382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:06.139063 12382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:06.143733 12382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:06.144060 12382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:06.164062 12382 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44105
I20260927 22:20:06.164096 12448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44105 every 8 connection(s)
I20260927 22:20:06.165683 12382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:06.169513 12449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:06.172363 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12382
I20260927 22:20:06.172590 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:06.173112 24245 external_mini_cluster.cc:1428] Setting key a1486ac04582707dc7435823c79de6a7
I20260927 22:20:06.181394 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:06.175026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36801 (local address 127.0.0.1:44105)
0927 22:20:06.175848 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:06.175860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:06.175904 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:20:06.177704 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:06.177728 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:06.178201 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:06.178564 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:06.179670 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:06.179677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:06.179691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:06.179705 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:06.179769 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:06.179790 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:06.180058 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:06.180209 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:20:06.180407 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":390,"threads_started":1}
I20260927 22:20:06.181983 12449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710: Bootstrap starting.
I20260927 22:20:06.185367 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:06.185395 12449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:06.186422 12449 log.cc:824] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:06.189010 12449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710: No bootstrap required, opened a new log
I20260927 22:20:06.192595 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:06.190039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58527 (local address 127.0.0.1:44105)
0927 22:20:06.190179 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:06.190183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:06.190553 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:20:06.190754 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:06.190758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:06.190800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:06.190883 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:06.191835 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:06.191839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:06.191843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:06.191864 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:06.191913 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:06.191934 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:06.192041 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:06.192155 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:20:06.192203 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 22:20:06.197239 12449 raft_consensus.cc:348] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } }
I20260927 22:20:06.197593 12449 raft_consensus.cc:374] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:06.197695 12449 raft_consensus.cc:729] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da43cec314144605a71238a110922710, State: Initialized, Role: FOLLOWER
I20260927 22:20:06.198279 12449 consensus_queue.cc:260] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } }
I20260927 22:20:06.198529 12449 raft_consensus.cc:388] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:06.198694 12449 raft_consensus.cc:482] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:06.198899 12449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:06.201058 12449 raft_consensus.cc:504] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } }
I20260927 22:20:06.201527 12449 leader_election.cc:302] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da43cec314144605a71238a110922710; no voters:
I20260927 22:20:06.202131 12449 leader_election.cc:288] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:06.202308 12459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:06.202977 12459 raft_consensus.cc:686] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [term 1 LEADER]: Becoming Leader. State: Replica: da43cec314144605a71238a110922710, State: Running, Role: LEADER
I20260927 22:20:06.203469 12459 consensus_queue.cc:237] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } }
I20260927 22:20:06.204178 12449 sys_catalog.cc:564] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:06.206431 12461 sys_catalog.cc:455] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da43cec314144605a71238a110922710. Latest consensus state: current_term: 1 leader_uuid: "da43cec314144605a71238a110922710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } } }
I20260927 22:20:06.206862 12461 sys_catalog.cc:458] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:06.206714 12460 sys_catalog.cc:455] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da43cec314144605a71238a110922710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da43cec314144605a71238a110922710" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44105 } } }
I20260927 22:20:06.206979 12460 sys_catalog.cc:458] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:06.209515 12466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:06.214326 12466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:06.222875 12466 catalog_manager.cc:1347] Generated new cluster ID: 397ac6a874ae423d8d6cca9b5ea46fc6
I20260927 22:20:06.222982 12466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:06.243867 12466 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:06.244334 12466 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:06.261924 12466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da43cec314144605a71238a110922710: Generated new TSK 0
I20260927 22:20:06.262935 12466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:06.276078 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12382
W20260927 22:20:06.277245 12444 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36801
W20260927 22:20:06.278069 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:06.278234 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12382
2026-09-27T22:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:06 dist-test-slave-3wc3 krb5kdc[12480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:06 dist-test-slave-3wc3 krb5kdc[12480](info): set up 2 sockets
Sep 27 22:20:06 dist-test-slave-3wc3 krb5kdc[12480](info): commencing operation
krb5kdc: starting...
W20260927 22:20:07.382359 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:07 dist-test-slave-3wc3 krb5kdc[12480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:07.640875 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:07.976377 12493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:07.976667 12493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:07.976847 12493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:07.984362 12493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:07.984479 12493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:07.984606 12493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:07.984717 12493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:07.984783 12493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:07.994616 12493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:07.995944 12493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:07.997254 12493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:08.008940 12499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:08.008951 12501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:08.008996 12498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:08.009721 12493 server_base.cc:1034] running on GCE node
I20260927 22:20:08.010505 12493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:08.011845 12493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:08.013082 12493 hybrid_clock.cc:648] HybridClock initialized: now 1790547608013033 us; error 59 us; skew 500 ppm
I20260927 22:20:08.013412 12493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:08 dist-test-slave-3wc3 krb5kdc[12480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:08.016661 12493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:08.018077 12493 webserver.cc:460] Webserver started at http://127.0.0.1:40191/ using document root <none> and password file <none>
I20260927 22:20:08.018672 12493 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:08.018769 12493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:08.018999 12493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:08.021739 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2"
format_stamp: "Formatted at 2026-09-27 22:20:08 on dist-test-slave-3wc3"
server_key: "5e400b0904068809ca2a237b6bd38ff5"
server_key_iv: "a62a8efb520bfddb5ccfcb42cbdb30ba"
server_key_version: "encryptionkey@0"
I20260927 22:20:08.022495 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2"
format_stamp: "Formatted at 2026-09-27 22:20:08 on dist-test-slave-3wc3"
server_key: "5e400b0904068809ca2a237b6bd38ff5"
server_key_iv: "a62a8efb520bfddb5ccfcb42cbdb30ba"
server_key_version: "encryptionkey@0"
I20260927 22:20:08.027221 12493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:20:08.030493 12508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:08.031737 12493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:20:08.031917 12493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2"
format_stamp: "Formatted at 2026-09-27 22:20:08 on dist-test-slave-3wc3"
server_key: "5e400b0904068809ca2a237b6bd38ff5"
server_key_iv: "a62a8efb520bfddb5ccfcb42cbdb30ba"
server_key_version: "encryptionkey@0"
I20260927 22:20:08.032107 12493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:08.051975 12493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:08.056569 12493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:08.056885 12493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:08.076434 12493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37365
I20260927 22:20:08.076507 12559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37365 every 8 connection(s)
I20260927 22:20:08.077809 12493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:08.081475 12560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:08.087024 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12493
I20260927 22:20:08.087210 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:08.087755 24245 external_mini_cluster.cc:1428] Setting key 746a21232e2ca223e000095141f9a5df
I20260927 22:20:08.092626 12560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2: Bootstrap starting.
I20260927 22:20:08.095775 12560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:08.096784 12560 log.cc:824] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:08.099586 12560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2: No bootstrap required, opened a new log
Sep 27 22:20:08 dist-test-slave-3wc3 krb5kdc[12480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:20:08.107200 12560 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } }
I20260927 22:20:08.107589 12560 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:08.107722 12560 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e7d3817a0641b69bf0fa56f2ae30c2, State: Initialized, Role: FOLLOWER
I20260927 22:20:08.108383 12560 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } }
I20260927 22:20:08.108652 12560 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:08.108814 12560 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:08.109020 12560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:08.110245 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:08.089642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39039 (local address 127.0.0.1:37365)
0927 22:20:08.090483 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:08.090497 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:08.090537 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:20:08.092159 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:08.092181 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:08.092574 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:08.093005 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:08.094511 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:08.095795 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:08.097136 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:08.098257 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:08.098455 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:08.102484 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:08.102495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:08.102509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:08.102558 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:08.105731 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:08.106496 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:08.106507 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:08.106520 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:08.106688 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:08.107078 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:08.107089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:08.107092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:08.107659 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:08.107972 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:08.108409 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 22:20:08.109061 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":379,"threads_started":1}
I20260927 22:20:08.112093 12560 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } }
I20260927 22:20:08.112659 12560 leader_election.cc:302] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e7d3817a0641b69bf0fa56f2ae30c2; no voters:
I20260927 22:20:08.113346 12560 leader_election.cc:288] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:08.113494 12565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:08.114573 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:08.114631 12565 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [term 1 LEADER]: Becoming Leader. State: Replica: f0e7d3817a0641b69bf0fa56f2ae30c2, State: Running, Role: LEADER
I20260927 22:20:08.115270 12565 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } }
I20260927 22:20:08.116940 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e7d3817a0641b69bf0fa56f2ae30c2. Latest consensus state: current_term: 1 leader_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } } }
I20260927 22:20:08.117274 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:08.117758 12567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e7d3817a0641b69bf0fa56f2ae30c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37365 } } }
I20260927 22:20:08.118170 12567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:08.129619 12560 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:08.129845 12569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:08.134868 12569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:08.145470 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:08.127874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46675 (local address 127.0.0.1:37365)
0927 22:20:08.128059 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:08.128069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:08.128650 (+ 581us) server_negotiation.cc:407] Connection header received
0927 22:20:08.128815 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:08.128821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:08.128899 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:08.129003 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:08.130933 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:08.132154 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:08.133927 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:08.134647 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:08.134797 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:08.137927 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:08.137942 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:08.137948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:08.138016 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:08.141101 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:08.141791 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:08.141796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:08.141800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:08.141920 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:08.142404 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:08.142412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:08.142415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:08.142819 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:08.143072 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:08.144638 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0927 22:20:08.144918 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:20:08.152247 12569 catalog_manager.cc:1347] Generated new cluster ID: b427d0a0b47240318ecf932cb02bbf88
I20260927 22:20:08.152356 12569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:08.162508 12569 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:08.163818 12569 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:08.173574 12569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e7d3817a0641b69bf0fa56f2ae30c2: Generated new TSK 0
I20260927 22:20:08.174576 12569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:08.182643 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12493
I20260927 22:20:08.183766 12555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:08.271247 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12493
2026-09-27T22:20:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:08 dist-test-slave-3wc3 krb5kdc[12592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:08 dist-test-slave-3wc3 krb5kdc[12592](info): set up 2 sockets
Sep 27 22:20:08 dist-test-slave-3wc3 krb5kdc[12592](info): commencing operation
krb5kdc: starting...
W20260927 22:20:09.414393 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:09 dist-test-slave-3wc3 krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:09.670461 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:10.013247 12605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:10.013532 12605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:10.013741 12605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:10.021565 12605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:10.021675 12605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:10.021780 12605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:10.021888 12605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:10.022009 12605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:10.031965 12605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:10.033257 12605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:10.034581 12605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:10.046196 12610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:10.046171 12611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:10.046329 12613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:10.047694 12605 server_base.cc:1034] running on GCE node
I20260927 22:20:10.048413 12605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:10.049916 12605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:10.051205 12605 hybrid_clock.cc:648] HybridClock initialized: now 1790547610051153 us; error 32 us; skew 500 ppm
I20260927 22:20:10.051707 12605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:10 dist-test-slave-3wc3 krb5kdc[12592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:10.056329 12605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:10.058117 12605 webserver.cc:460] Webserver started at http://127.0.0.1:36467/ using document root <none> and password file <none>
I20260927 22:20:10.058995 12605 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:10.059141 12605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:10.059435 12605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:10.062454 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "8f419fc2e046439a941b31d98f51e27b"
format_stamp: "Formatted at 2026-09-27 22:20:10 on dist-test-slave-3wc3"
server_key: "f6556a4c67c2e6b50fd7f5145170aa7d"
server_key_iv: "76e8df8a76bb651bd0ddd6caeab4cf03"
server_key_version: "encryptionkey@0"
I20260927 22:20:10.063225 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "8f419fc2e046439a941b31d98f51e27b"
format_stamp: "Formatted at 2026-09-27 22:20:10 on dist-test-slave-3wc3"
server_key: "f6556a4c67c2e6b50fd7f5145170aa7d"
server_key_iv: "76e8df8a76bb651bd0ddd6caeab4cf03"
server_key_version: "encryptionkey@0"
I20260927 22:20:10.068219 12605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:20:10.071666 12620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:10.072914 12605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:20:10.073118 12605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "8f419fc2e046439a941b31d98f51e27b"
format_stamp: "Formatted at 2026-09-27 22:20:10 on dist-test-slave-3wc3"
server_key: "f6556a4c67c2e6b50fd7f5145170aa7d"
server_key_iv: "76e8df8a76bb651bd0ddd6caeab4cf03"
server_key_version: "encryptionkey@0"
I20260927 22:20:10.073321 12605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:10.095089 12605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:10.099385 12605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:10.099679 12605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:10.118806 12605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42729
I20260927 22:20:10.118891 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42729 every 8 connection(s)
I20260927 22:20:10.120186 12605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:10.124367 12672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:10.125048 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12605
I20260927 22:20:10.125257 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:10.125726 24245 external_mini_cluster.cc:1428] Setting key dc7f40664de8cc9f25fddf3e7b5a8057
I20260927 22:20:10.138195 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b: Bootstrap starting.
I20260927 22:20:10.140189 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:10.127771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44365 (local address 127.0.0.1:42729)
0927 22:20:10.128582 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:10.128594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:10.128636 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:20:10.130604 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:10.130627 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:10.131056 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:10.131394 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:10.132140 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:10.133650 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:10.135585 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:10.136343 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:10.136494 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:10.137758 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:10.137766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:10.137780 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:10.138447 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:10.138539 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:10.138563 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:10.138654 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:10.138890 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:20:10.139159 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":377,"threads_started":1}
I20260927 22:20:10.141669 12672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:10.142967 12672 log.cc:824] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:10.144305 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:10.146016 12672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b: No bootstrap required, opened a new log
I20260927 22:20:10.155551 12672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260927 22:20:10.155929 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:10.148873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53683 (local address 127.0.0.1:42729)
0927 22:20:10.149054 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:10.149062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:10.149592 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:20:10.149774 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:10.149777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:10.149837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:10.149918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:10.150610 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:10.151446 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:10.153349 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:10.153854 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:10.154058 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:10.155155 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:10.155160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:10.155163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:10.155180 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:10.155218 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:10.155233 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:10.155309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:10.155470 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:20:10.155547 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:20:10.156049 12672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:10.156155 12672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f419fc2e046439a941b31d98f51e27b, State: Initialized, Role: FOLLOWER
I20260927 22:20:10.156824 12672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260927 22:20:10.157078 12672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:10.157296 12672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:10.157483 12672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:10.160051 12672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260927 22:20:10.160507 12672 leader_election.cc:302] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f419fc2e046439a941b31d98f51e27b; no voters:
I20260927 22:20:10.161088 12672 leader_election.cc:288] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:10.161252 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:10.161985 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [term 1 LEADER]: Becoming Leader. State: Replica: 8f419fc2e046439a941b31d98f51e27b, State: Running, Role: LEADER
I20260927 22:20:10.162541 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260927 22:20:10.163252 12672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:10.165423 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f419fc2e046439a941b31d98f51e27b. Latest consensus state: current_term: 1 leader_uuid: "8f419fc2e046439a941b31d98f51e27b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } } }
I20260927 22:20:10.165916 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:10.166214 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f419fc2e046439a941b31d98f51e27b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f419fc2e046439a941b31d98f51e27b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } } }
I20260927 22:20:10.166630 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:10.167976 12689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:10.172897 12689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:10.181129 12689 catalog_manager.cc:1347] Generated new cluster ID: 96a54072f8b94c77bba79803f3e96e5d
I20260927 22:20:10.181243 12689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:10.228523 12689 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:10.230368 12689 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:10.239220 12689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f419fc2e046439a941b31d98f51e27b: Generated new TSK 0
I20260927 22:20:10.240341 12689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:10.248888 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12605
W20260927 22:20:10.249907 12667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44365
W20260927 22:20:10.250626 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:10.250821 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12605
2026-09-27T22:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:10 dist-test-slave-3wc3 krb5kdc[12703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:10 dist-test-slave-3wc3 krb5kdc[12703](info): set up 2 sockets
Sep 27 22:20:10 dist-test-slave-3wc3 krb5kdc[12703](info): commencing operation
krb5kdc: starting...
W20260927 22:20:11.350337 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:11 dist-test-slave-3wc3 krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:11.609790 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:11.939126 12716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:11.939460 12716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:11.939630 12716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:11.946897 12716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:11.947046 12716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:11.947149 12716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:11.947253 12716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:11.947327 12716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:11.957007 12716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:11.958300 12716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:11.959645 12716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:11.971918 12721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:11.971944 12724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:11.972002 12722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:11.972699 12716 server_base.cc:1034] running on GCE node
I20260927 22:20:11.973464 12716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:11.974790 12716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:11.976032 12716 hybrid_clock.cc:648] HybridClock initialized: now 1790547611975978 us; error 45 us; skew 500 ppm
I20260927 22:20:11.976440 12716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:11 dist-test-slave-3wc3 krb5kdc[12703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:11.983202 12716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:11.984609 12716 webserver.cc:460] Webserver started at http://127.0.0.1:41797/ using document root <none> and password file <none>
I20260927 22:20:11.985287 12716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:11.985430 12716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:11.985733 12716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:11.988744 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "9d7e5b3346e449beb016bfeb3452fbc0"
format_stamp: "Formatted at 2026-09-27 22:20:11 on dist-test-slave-3wc3"
server_key: "36fb41fe3ef427c5b85121abac27cd4c"
server_key_iv: "8721da06f02774ab7281fffe2d83d343"
server_key_version: "encryptionkey@0"
I20260927 22:20:11.989501 12716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "9d7e5b3346e449beb016bfeb3452fbc0"
format_stamp: "Formatted at 2026-09-27 22:20:11 on dist-test-slave-3wc3"
server_key: "36fb41fe3ef427c5b85121abac27cd4c"
server_key_iv: "8721da06f02774ab7281fffe2d83d343"
server_key_version: "encryptionkey@0"
I20260927 22:20:11.994638 12716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:20:11.997927 12731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:11.999356 12716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:11.999571 12716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "9d7e5b3346e449beb016bfeb3452fbc0"
format_stamp: "Formatted at 2026-09-27 22:20:11 on dist-test-slave-3wc3"
server_key: "36fb41fe3ef427c5b85121abac27cd4c"
server_key_iv: "8721da06f02774ab7281fffe2d83d343"
server_key_version: "encryptionkey@0"
I20260927 22:20:11.999774 12716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:12.005594 12716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:12.009716 12716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:12.010102 12716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:12.028806 12716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40155
I20260927 22:20:12.028846 12782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40155 every 8 connection(s)
I20260927 22:20:12.030210 12716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:12.033799 12783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:12.034276 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12716
I20260927 22:20:12.034474 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:12.034929 24245 external_mini_cluster.cc:1428] Setting key 1cd16bd414de0def927b0b81860de766
I20260927 22:20:12.044186 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:12.036696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33545 (local address 127.0.0.1:40155)
0927 22:20:12.037538 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:12.037553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:12.037612 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:20:12.040407 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:12.040436 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:12.040899 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:12.041351 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:12.042317 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:12.042325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:12.042343 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:12.042358 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:12.042422 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:12.042451 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:12.042669 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:12.042790 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:20:12.042977 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
I20260927 22:20:12.047865 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:12.048111 12783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0: Bootstrap starting.
I20260927 22:20:12.051297 12783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:12.052764 12783 log.cc:824] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:12.056819 12783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0: No bootstrap required, opened a new log
I20260927 22:20:12.056898 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:12.052633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59267 (local address 127.0.0.1:40155)
0927 22:20:12.053734 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:12.053742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:12.054388 (+ 646us) server_negotiation.cc:407] Connection header received
0927 22:20:12.054588 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:12.054593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:12.054669 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:12.054796 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:12.055995 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:12.056003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:12.056009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:12.056030 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:12.056083 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:12.056103 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:12.056191 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:12.056437 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:20:12.056524 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959}
I20260927 22:20:12.066954 12783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } }
I20260927 22:20:12.067386 12783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:12.067514 12783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7e5b3346e449beb016bfeb3452fbc0, State: Initialized, Role: FOLLOWER
I20260927 22:20:12.068096 12783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } }
I20260927 22:20:12.068346 12783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:12.068503 12783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:12.068681 12783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:12.071071 12783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } }
I20260927 22:20:12.071647 12783 leader_election.cc:302] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d7e5b3346e449beb016bfeb3452fbc0; no voters:
I20260927 22:20:12.072331 12783 leader_election.cc:288] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:12.072453 12793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:12.073297 12793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7e5b3346e449beb016bfeb3452fbc0, State: Running, Role: LEADER
I20260927 22:20:12.073813 12793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } }
I20260927 22:20:12.074690 12783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:12.076681 12795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d7e5b3346e449beb016bfeb3452fbc0. Latest consensus state: current_term: 1 leader_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } } }
I20260927 22:20:12.077162 12795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:12.077078 12794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7e5b3346e449beb016bfeb3452fbc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40155 } } }
I20260927 22:20:12.077385 12794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:12.081401 12801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:12.085938 12801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:12.094491 12801 catalog_manager.cc:1347] Generated new cluster ID: 7f6d2d8d41c6474a982b47f3ff36b39a
I20260927 22:20:12.094604 12801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:12.109122 12801 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:12.109557 12801 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:12.120271 12801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d7e5b3346e449beb016bfeb3452fbc0: Generated new TSK 0
I20260927 22:20:12.121194 12801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:12.135486 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12716
W20260927 22:20:12.136848 12778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33545
W20260927 22:20:12.137596 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:12.137745 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12716
2026-09-27T22:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:12 dist-test-slave-3wc3 krb5kdc[12814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:12 dist-test-slave-3wc3 krb5kdc[12814](info): set up 2 sockets
Sep 27 22:20:12 dist-test-slave-3wc3 krb5kdc[12814](info): commencing operation
krb5kdc: starting...
W20260927 22:20:13.234470 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:13 dist-test-slave-3wc3 krb5kdc[12814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:13.501286 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:13.845516 12827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:13.845870 12827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:13.846058 12827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:13.853394 12827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:13.853561 12827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:13.853677 12827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:13.853787 12827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:13.853865 12827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:13.864470 12827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:13.865828 12827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:13.867230 12827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:13.880142 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:13.880281 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:13.880192 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:13.881193 12827 server_base.cc:1034] running on GCE node
I20260927 22:20:13.882040 12827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:13.883483 12827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:13.884711 12827 hybrid_clock.cc:648] HybridClock initialized: now 1790547613884676 us; error 39 us; skew 500 ppm
I20260927 22:20:13.885066 12827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:13 dist-test-slave-3wc3 krb5kdc[12814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:13.888237 12827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:13.889613 12827 webserver.cc:460] Webserver started at http://127.0.0.1:40341/ using document root <none> and password file <none>
I20260927 22:20:13.890311 12827 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:13.890413 12827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:13.890673 12827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:13.893530 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "a90a1e9ef1944913abacbd5c7fab0683"
format_stamp: "Formatted at 2026-09-27 22:20:13 on dist-test-slave-3wc3"
server_key: "49ea9a29d9d1ce76a7ed49adcadd890a"
server_key_iv: "3dd9efc4437b9edeff836f93fa86f2e2"
server_key_version: "encryptionkey@0"
I20260927 22:20:13.894299 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "a90a1e9ef1944913abacbd5c7fab0683"
format_stamp: "Formatted at 2026-09-27 22:20:13 on dist-test-slave-3wc3"
server_key: "49ea9a29d9d1ce76a7ed49adcadd890a"
server_key_iv: "3dd9efc4437b9edeff836f93fa86f2e2"
server_key_version: "encryptionkey@0"
I20260927 22:20:13.899344 12827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:20:13.902707 12842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:13.904121 12827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:13.904318 12827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "a90a1e9ef1944913abacbd5c7fab0683"
format_stamp: "Formatted at 2026-09-27 22:20:13 on dist-test-slave-3wc3"
server_key: "49ea9a29d9d1ce76a7ed49adcadd890a"
server_key_iv: "3dd9efc4437b9edeff836f93fa86f2e2"
server_key_version: "encryptionkey@0"
I20260927 22:20:13.904520 12827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:13.935472 12827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:13.939749 12827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:13.940058 12827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:13.959218 12827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45771
I20260927 22:20:13.959249 12893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45771 every 8 connection(s)
I20260927 22:20:13.960592 12827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:13.964365 12894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:13.967001 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12827
I20260927 22:20:13.967191 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:13.967635 24245 external_mini_cluster.cc:1428] Setting key 63c0b003f3fbe45c8dc76387e0f7a320
I20260927 22:20:13.977594 12894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683: Bootstrap starting.
I20260927 22:20:13.981721 12894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:13.983225 12894 log.cc:824] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:20:13 dist-test-slave-3wc3 krb5kdc[12814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:20:13.986773 12894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683: No bootstrap required, opened a new log
I20260927 22:20:13.993564 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:13.969397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:38269 (local address 127.0.0.1:45771)
0927 22:20:13.970304 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:13.970326 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:13.970388 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:20:13.972871 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:13.972904 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:13.973491 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:13.973834 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:13.975788 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:13.977141 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:13.979042 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:13.979763 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:13.979906 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:13.985483 (+ 5577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:13.985500 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:13.985519 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:13.985560 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:13.989101 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:13.990030 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:13.990036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:13.990042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:13.990210 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:13.990722 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:13.990726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:13.990728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:13.991278 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:13.991640 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:13.992146 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:20:13.992490 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":401,"threads_started":1}
I20260927 22:20:13.995092 12894 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } }
I20260927 22:20:13.995496 12894 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:13.995623 12894 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90a1e9ef1944913abacbd5c7fab0683, State: Initialized, Role: FOLLOWER
I20260927 22:20:13.996232 12894 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } }
I20260927 22:20:13.996517 12894 raft_consensus.cc:388] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:13.996658 12894 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:13.996888 12894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:13.997840 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:13.999366 12894 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } }
I20260927 22:20:13.999866 12894 leader_election.cc:302] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a90a1e9ef1944913abacbd5c7fab0683; no voters:
I20260927 22:20:14.000532 12894 leader_election.cc:288] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:14.000655 12900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:14.003139 12894 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:14.008230 12900 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [term 1 LEADER]: Becoming Leader. State: Replica: a90a1e9ef1944913abacbd5c7fab0683, State: Running, Role: LEADER
I20260927 22:20:14.009076 12900 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } }
I20260927 22:20:14.012892 12903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90a1e9ef1944913abacbd5c7fab0683. Latest consensus state: current_term: 1 leader_uuid: "a90a1e9ef1944913abacbd5c7fab0683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } } }
I20260927 22:20:14.013209 12903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:14.013744 12902 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a90a1e9ef1944913abacbd5c7fab0683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90a1e9ef1944913abacbd5c7fab0683" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45771 } } }
I20260927 22:20:14.014102 12902 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:14.014297 12913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:14.028270 12913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:14.032222 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:14.008565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:33635 (local address 127.0.0.1:45771)
0927 22:20:14.008860 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:14.008866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:14.009554 (+ 688us) server_negotiation.cc:407] Connection header received
0927 22:20:14.009831 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:14.009837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:14.009910 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:14.010084 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:14.012107 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:14.013512 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:14.015792 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:14.018821 (+ 3029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:14.019016 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:14.023495 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:14.023505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:14.023513 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:14.023568 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:14.027984 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:14.029055 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:14.029061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:14.029065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:14.029220 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:14.029807 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:14.029814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:14.029822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:14.030384 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:14.030721 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:14.031600 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 22:20:14.031822 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 22:20:14.040688 12913 catalog_manager.cc:1347] Generated new cluster ID: 5cdbd27841474148a4f779060d53e50c
I20260927 22:20:14.040791 12913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:14.065058 12913 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:14.066993 12913 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:14.085589 12913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a90a1e9ef1944913abacbd5c7fab0683: Generated new TSK 0
I20260927 22:20:14.086500 12913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:14.100883 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12827
I20260927 22:20:14.102041 12889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:14.195673 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12827
W20260927 22:20:14.212788 12812 connection.cc:535] client connection to 127.0.0.1:45771 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45771): Connection reset by peer (error 104)
2026-09-27T22:20:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2057 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:14 dist-test-slave-3wc3 krb5kdc[12926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:14 dist-test-slave-3wc3 krb5kdc[12926](info): set up 2 sockets
Sep 27 22:20:14 dist-test-slave-3wc3 krb5kdc[12926](info): commencing operation
krb5kdc: starting...
W20260927 22:20:15.298610 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:15 dist-test-slave-3wc3 krb5kdc[12926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:15.553396 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:15.879859 12939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:15.880215 12939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:15.880390 12939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:15.887511 12939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:15.887674 12939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:15.887784 12939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:15.887902 12939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:15.887991 12939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:15.897495 12939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:15.899113 12939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:15.900475 12939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:15.912153 12945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:15.912230 12944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:15.912254 12947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:15.912828 12939 server_base.cc:1034] running on GCE node
I20260927 22:20:15.913576 12939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:15.915033 12939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:15.916262 12939 hybrid_clock.cc:648] HybridClock initialized: now 1790547615916231 us; error 55 us; skew 500 ppm
I20260927 22:20:15.916602 12939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:15 dist-test-slave-3wc3 krb5kdc[12926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:15.919708 12939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:15.921247 12939 webserver.cc:460] Webserver started at http://127.0.0.1:42611/ using document root <none> and password file <none>
I20260927 22:20:15.921929 12939 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:15.922128 12939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:15.922417 12939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:15.925308 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "8d9b143d99b44b949d53775594de92c0"
format_stamp: "Formatted at 2026-09-27 22:20:15 on dist-test-slave-3wc3"
server_key: "9ceee4794fd19607b43ee6cfabdf4d2c"
server_key_iv: "8e503552ddaf4ce7e214df1c2ee10534"
server_key_version: "encryptionkey@0"
I20260927 22:20:15.926173 12939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "8d9b143d99b44b949d53775594de92c0"
format_stamp: "Formatted at 2026-09-27 22:20:15 on dist-test-slave-3wc3"
server_key: "9ceee4794fd19607b43ee6cfabdf4d2c"
server_key_iv: "8e503552ddaf4ce7e214df1c2ee10534"
server_key_version: "encryptionkey@0"
I20260927 22:20:15.931324 12939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:20:15.934741 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:15.935963 12939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:15.936142 12939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "8d9b143d99b44b949d53775594de92c0"
format_stamp: "Formatted at 2026-09-27 22:20:15 on dist-test-slave-3wc3"
server_key: "9ceee4794fd19607b43ee6cfabdf4d2c"
server_key_iv: "8e503552ddaf4ce7e214df1c2ee10534"
server_key_version: "encryptionkey@0"
I20260927 22:20:15.936331 12939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:15.955458 12939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:15.959896 12939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:15.960208 12939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:15.979458 12939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40429
I20260927 22:20:15.979488 13005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40429 every 8 connection(s)
I20260927 22:20:15.981359 12939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:15.985201 13006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:15.988670 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 12939
I20260927 22:20:15.988826 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:15.989301 24245 external_mini_cluster.cc:1428] Setting key b6c4ce5365fbbc2d9e14cce581f56706
I20260927 22:20:15.997422 13006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0: Bootstrap starting.
I20260927 22:20:16.001116 13006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:16.002421 13006 log.cc:824] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:16.002532 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:15.991060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:51329 (local address 127.0.0.1:40429)
0927 22:20:15.991829 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:15.991841 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:15.991882 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:20:15.993526 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:15.993550 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:15.994033 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:15.994382 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:15.994971 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:15.996244 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:15.998050 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:15.998691 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:15.998811 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:16.000197 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:16.000205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:16.000218 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:16.000754 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:16.000829 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:16.000853 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:16.000968 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:16.001335 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 22:20:16.001468 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":344,"threads_started":1}
I20260927 22:20:16.005368 13006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0: No bootstrap required, opened a new log
I20260927 22:20:16.006407 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:16.013314 13006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } }
I20260927 22:20:16.013713 13006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:16.013823 13006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9b143d99b44b949d53775594de92c0, State: Initialized, Role: FOLLOWER
I20260927 22:20:16.014420 13006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } }
I20260927 22:20:16.014674 13006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:16.014832 13006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:16.015059 13006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:16.017326 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:16.010803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:56677 (local address 127.0.0.1:40429)
0927 22:20:16.011010 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:16.011017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:16.011377 (+ 360us) server_negotiation.cc:407] Connection header received
0927 22:20:16.011596 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:16.011600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:16.011663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:16.011774 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:16.012273 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:16.013017 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:16.014589 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:16.015269 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:16.015419 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:16.016558 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:16.016564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:16.016568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:16.016596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:16.016650 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:16.016666 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:16.016790 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:16.016987 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 22:20:16.017037 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:20:16.017822 13006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } }
I20260927 22:20:16.018451 13006 leader_election.cc:302] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d9b143d99b44b949d53775594de92c0; no voters:
I20260927 22:20:16.019109 13006 leader_election.cc:288] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:16.019171 13016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:16.019887 13016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d9b143d99b44b949d53775594de92c0, State: Running, Role: LEADER
I20260927 22:20:16.020426 13016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } }
I20260927 22:20:16.021183 13006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:16.023356 13018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9b143d99b44b949d53775594de92c0. Latest consensus state: current_term: 1 leader_uuid: "8d9b143d99b44b949d53775594de92c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } } }
I20260927 22:20:16.023918 13018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:16.023682 13017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d9b143d99b44b949d53775594de92c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9b143d99b44b949d53775594de92c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40429 } } }
I20260927 22:20:16.024081 13017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:16.032424 13023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:16.036442 13023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:16.044868 13023 catalog_manager.cc:1347] Generated new cluster ID: 8a7fa26bd0e94d26b4851af1ef909145
I20260927 22:20:16.044979 13023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:16.094548 13023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:16.096427 13023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:16.107457 13023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d9b143d99b44b949d53775594de92c0: Generated new TSK 0
I20260927 22:20:16.108479 13023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:16.118546 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12939
W20260927 22:20:16.119820 13001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:51329
W20260927 22:20:16.120563 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 12939: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:16.120718 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 12939
W20260927 22:20:16.137840 12924 connection.cc:535] client connection to 127.0.0.1:40429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40429): Connection reset by peer (error 104)
2026-09-27T22:20:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:16 dist-test-slave-3wc3 krb5kdc[13037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:16 dist-test-slave-3wc3 krb5kdc[13037](info): set up 2 sockets
Sep 27 22:20:16 dist-test-slave-3wc3 krb5kdc[13037](info): commencing operation
krb5kdc: starting...
W20260927 22:20:17.278429 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:17 dist-test-slave-3wc3 krb5kdc[13037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:17.533915 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:17.860946 13050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:17.861222 13050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:17.861315 13050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:17.868296 13050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:17.868392 13050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:17.868450 13050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:17.868510 13050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:17.868552 13050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:17.878135 13050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:17.879348 13050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:17.880594 13050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:17.892319 13056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:17.892520 13055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:17.892912 13058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:17.893301 13050 server_base.cc:1034] running on GCE node
I20260927 22:20:17.894101 13050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:17.895365 13050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:17.896603 13050 hybrid_clock.cc:648] HybridClock initialized: now 1790547617896571 us; error 53 us; skew 500 ppm
I20260927 22:20:17.896948 13050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:17 dist-test-slave-3wc3 krb5kdc[13037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:17.906134 13050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:17.907855 13050 webserver.cc:460] Webserver started at http://127.0.0.1:46369/ using document root <none> and password file <none>
I20260927 22:20:17.908564 13050 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:17.908723 13050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:17.909042 13050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:17.912099 13050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "15b58ee2d88340fd98ca9c22277494e3"
format_stamp: "Formatted at 2026-09-27 22:20:17 on dist-test-slave-3wc3"
server_key: "a412bfc7629780e452c17c159fb681d8"
server_key_iv: "a6be8575d72af796dfc30cf8a4cfb6c4"
server_key_version: "encryptionkey@0"
I20260927 22:20:17.912879 13050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "15b58ee2d88340fd98ca9c22277494e3"
format_stamp: "Formatted at 2026-09-27 22:20:17 on dist-test-slave-3wc3"
server_key: "a412bfc7629780e452c17c159fb681d8"
server_key_iv: "a6be8575d72af796dfc30cf8a4cfb6c4"
server_key_version: "encryptionkey@0"
I20260927 22:20:17.918242 13050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:20:17.921754 13065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:17.923103 13050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:20:17.923282 13050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "15b58ee2d88340fd98ca9c22277494e3"
format_stamp: "Formatted at 2026-09-27 22:20:17 on dist-test-slave-3wc3"
server_key: "a412bfc7629780e452c17c159fb681d8"
server_key_iv: "a6be8575d72af796dfc30cf8a4cfb6c4"
server_key_version: "encryptionkey@0"
I20260927 22:20:17.923502 13050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:17.929126 13050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:17.933351 13050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:17.933655 13050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:17.951694 13050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40589
I20260927 22:20:17.951727 13116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40589 every 8 connection(s)
I20260927 22:20:17.953084 13050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:17.956712 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:17.957468 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13050
I20260927 22:20:17.957660 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:17.958117 24245 external_mini_cluster.cc:1428] Setting key 8e3895ed48bdaace78eb563fb59cabf2
I20260927 22:20:17.965731 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:17.959967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:33911 (local address 127.0.0.1:40589)
0927 22:20:17.960760 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:17.960770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:17.960814 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:20:17.962562 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:17.962585 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:17.962926 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:17.963242 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:17.964294 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:17.964301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:17.964313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:17.964326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:17.964380 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:17.964402 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:17.964561 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:17.964677 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:20:17.964873 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":357,"threads_started":1}
I20260927 22:20:17.969223 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:17.969292 13117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3: Bootstrap starting.
I20260927 22:20:17.972272 13117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:17.973332 13117 log.cc:824] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:17.976112 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:17.973598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:57461 (local address 127.0.0.1:40589)
0927 22:20:17.973791 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:17.973799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:17.974257 (+ 458us) server_negotiation.cc:407] Connection header received
0927 22:20:17.974420 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:17.974423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:17.974471 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:17.974554 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:17.975494 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:17.975499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:17.975503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:17.975521 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:17.975561 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:17.975584 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:17.975709 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:17.975778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:20:17.975830 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:20:17.976464 13117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3: No bootstrap required, opened a new log
I20260927 22:20:17.983393 13117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } }
I20260927 22:20:17.983765 13117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:17.983877 13117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b58ee2d88340fd98ca9c22277494e3, State: Initialized, Role: FOLLOWER
I20260927 22:20:17.984405 13117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } }
I20260927 22:20:17.984635 13117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:17.984782 13117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:17.984937 13117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:17.987823 13117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } }
I20260927 22:20:17.988365 13117 leader_election.cc:302] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b58ee2d88340fd98ca9c22277494e3; no voters:
I20260927 22:20:17.989341 13117 leader_election.cc:288] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:17.989477 13127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:17.990170 13127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [term 1 LEADER]: Becoming Leader. State: Replica: 15b58ee2d88340fd98ca9c22277494e3, State: Running, Role: LEADER
I20260927 22:20:17.990749 13127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } }
I20260927 22:20:17.991371 13117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:17.993472 13129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b58ee2d88340fd98ca9c22277494e3. Latest consensus state: current_term: 1 leader_uuid: "15b58ee2d88340fd98ca9c22277494e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } } }
I20260927 22:20:17.993499 13128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b58ee2d88340fd98ca9c22277494e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b58ee2d88340fd98ca9c22277494e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40589 } } }
I20260927 22:20:17.994048 13129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:17.994059 13128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:18.002527 13132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:18.009498 13132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:18.017927 13132 catalog_manager.cc:1347] Generated new cluster ID: cb85cf77a4de460cbbbd36e1cee2fbd4
I20260927 22:20:18.018124 13132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:18.033223 13132 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:18.033708 13132 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:18.047917 13132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b58ee2d88340fd98ca9c22277494e3: Generated new TSK 0
I20260927 22:20:18.049038 13132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:18.057401 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13050
W20260927 22:20:18.058563 13112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:33911
W20260927 22:20:18.059407 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:18.059556 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13050
2026-09-27T22:20:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:18 dist-test-slave-3wc3 krb5kdc[13148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:18 dist-test-slave-3wc3 krb5kdc[13148](info): set up 2 sockets
Sep 27 22:20:18 dist-test-slave-3wc3 krb5kdc[13148](info): commencing operation
krb5kdc: starting...
W20260927 22:20:19.178480 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:19 dist-test-slave-3wc3 krb5kdc[13148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:19.433460 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:19.764834 13161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:19.765170 13161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:19.765342 13161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:19.772465 13161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:19.772671 13161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:19.772780 13161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:19.772895 13161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:19.772984 13161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:19.782603 13161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:19.783931 13161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:19.785236 13161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:19.797132 13166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:19.797206 13169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:19.797224 13167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:19.798026 13161 server_base.cc:1034] running on GCE node
I20260927 22:20:19.798827 13161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:19.800323 13161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:19.801604 13161 hybrid_clock.cc:648] HybridClock initialized: now 1790547619801558 us; error 59 us; skew 500 ppm
I20260927 22:20:19.802021 13161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:19 dist-test-slave-3wc3 krb5kdc[13148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:19.805176 13161 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:19.806668 13161 webserver.cc:460] Webserver started at http://127.0.0.1:42313/ using document root <none> and password file <none>
I20260927 22:20:19.807339 13161 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:19.807574 13161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:19.807874 13161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:19.810855 13161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "e9d6a62ae9234908ab9ed23a2505bbeb"
format_stamp: "Formatted at 2026-09-27 22:20:19 on dist-test-slave-3wc3"
server_key: "020e713603993431bd6ba3c347f7e9b4"
server_key_iv: "825c80abf360c8a96fefb17b89986849"
server_key_version: "encryptionkey@0"
I20260927 22:20:19.811559 13161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "e9d6a62ae9234908ab9ed23a2505bbeb"
format_stamp: "Formatted at 2026-09-27 22:20:19 on dist-test-slave-3wc3"
server_key: "020e713603993431bd6ba3c347f7e9b4"
server_key_iv: "825c80abf360c8a96fefb17b89986849"
server_key_version: "encryptionkey@0"
I20260927 22:20:19.816660 13161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:20:19.820165 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:19.821436 13161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:19.821653 13161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "e9d6a62ae9234908ab9ed23a2505bbeb"
format_stamp: "Formatted at 2026-09-27 22:20:19 on dist-test-slave-3wc3"
server_key: "020e713603993431bd6ba3c347f7e9b4"
server_key_iv: "825c80abf360c8a96fefb17b89986849"
server_key_version: "encryptionkey@0"
I20260927 22:20:19.821849 13161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:19.862270 13161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:19.866645 13161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:19.866948 13161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:19.886052 13161 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45879
I20260927 22:20:19.886058 13227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45879 every 8 connection(s)
I20260927 22:20:19.887521 13161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:19.889153 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13161
I20260927 22:20:19.889353 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:19.889817 24245 external_mini_cluster.cc:1428] Setting key 28245b1c29b31e1b974189e96dddc39e
I20260927 22:20:19.891830 13228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:20:19 dist-test-slave-3wc3 krb5kdc[13148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:20:19.909332 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb: Bootstrap starting.
I20260927 22:20:19.914384 13228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:19.915907 13228 log.cc:824] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:19.918766 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:19.891792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:40191 (local address 127.0.0.1:45879)
0927 22:20:19.892588 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:19.892603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:19.892650 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:20:19.894856 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:19.894883 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:19.895410 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:19.895840 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:19.897141 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:19.899234 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:19.901230 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:19.902432 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:19.902714 (+ 282us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:19.908482 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:19.908492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:19.908513 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:19.908563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:19.913031 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:19.914125 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:19.914135 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:19.914145 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:19.914357 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:19.914930 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:19.914934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:19.914937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:19.915705 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:19.916022 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:19.917011 (+ 989us) server_negotiation.cc:299] Negotiation successful
0927 22:20:19.917415 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":309,"threads_started":1}
I20260927 22:20:19.919427 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb: No bootstrap required, opened a new log
I20260927 22:20:19.923262 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:19.931293 13228 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } }
I20260927 22:20:19.931735 13228 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:19.931910 13228 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d6a62ae9234908ab9ed23a2505bbeb, State: Initialized, Role: FOLLOWER
I20260927 22:20:19.932569 13228 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } }
I20260927 22:20:19.932845 13228 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:19.932979 13228 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:19.933153 13228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:19.936538 13228 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } }
I20260927 22:20:19.937171 13228 leader_election.cc:302] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d6a62ae9234908ab9ed23a2505bbeb; no voters:
I20260927 22:20:19.938658 13228 leader_election.cc:288] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:19.941000 13228 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:19.938822 13238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:19.941812 13238 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [term 1 LEADER]: Becoming Leader. State: Replica: e9d6a62ae9234908ab9ed23a2505bbeb, State: Running, Role: LEADER
I20260927 22:20:19.942543 13238 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } }
I20260927 22:20:19.946815 13240 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d6a62ae9234908ab9ed23a2505bbeb. Latest consensus state: current_term: 1 leader_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } } }
I20260927 22:20:19.947247 13240 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:19.948909 13244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:19.949091 13239 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d6a62ae9234908ab9ed23a2505bbeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45879 } } }
I20260927 22:20:19.949389 13239 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:19.951049 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:19.929434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:37695 (local address 127.0.0.1:45879)
0927 22:20:19.931262 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:19.931270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:19.931294 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:20:19.931347 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:19.931352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:19.931424 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:19.931529 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:19.933777 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:19.935011 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:19.936896 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:19.937708 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:19.937859 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:19.941179 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:19.941186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:19.941193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:19.941261 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:19.945639 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:19.946891 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:19.946900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:19.946906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:19.947226 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:19.947989 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:19.947997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:19.948017 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:19.948502 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:19.950027 (+ 1525us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:19.950259 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:20:19.950502 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1713}
I20260927 22:20:19.954330 13244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:19.969221 13244 catalog_manager.cc:1347] Generated new cluster ID: b205b110bb7649eea344fe1a9ae4eef8
I20260927 22:20:19.969339 13244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:19.979741 13244 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:19.981034 13244 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:19.990935 13244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d6a62ae9234908ab9ed23a2505bbeb: Generated new TSK 0
I20260927 22:20:19.991817 13244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:19.999495 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13161
I20260927 22:20:20.000916 13223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:20.094825 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13161
W20260927 22:20:20.112082 13146 connection.cc:535] client connection to 127.0.0.1:45879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45879): Connection reset by peer (error 104)
2026-09-27T22:20:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:20 dist-test-slave-3wc3 krb5kdc[13260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:20 dist-test-slave-3wc3 krb5kdc[13260](info): set up 2 sockets
Sep 27 22:20:20 dist-test-slave-3wc3 krb5kdc[13260](info): commencing operation
krb5kdc: starting...
W20260927 22:20:21.190420 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:21 dist-test-slave-3wc3 krb5kdc[13260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:21.453701 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:21.783619 13273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:21.783979 13273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:21.784173 13273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:21.791474 13273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:21.791652 13273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:21.791764 13273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:21.791863 13273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:21.791929 13273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:21.802047 13273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:21.803406 13273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:21.804768 13273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:21.817013 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:21.816929 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:21.816968 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:21.817755 13273 server_base.cc:1034] running on GCE node
I20260927 22:20:21.818547 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:21.819892 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:21.821116 13273 hybrid_clock.cc:648] HybridClock initialized: now 1790547621821077 us; error 51 us; skew 500 ppm
I20260927 22:20:21.821451 13273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:21 dist-test-slave-3wc3 krb5kdc[13260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:21.824434 13273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:21.825806 13273 webserver.cc:460] Webserver started at http://127.0.0.1:42957/ using document root <none> and password file <none>
I20260927 22:20:21.826502 13273 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:21.826646 13273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:21.826941 13273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:21.829810 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "c9e4102d05c04ec5b38db4f879dd4af6"
format_stamp: "Formatted at 2026-09-27 22:20:21 on dist-test-slave-3wc3"
server_key: "fc1a2ed69a046c48ebaab9f0dc729699"
server_key_iv: "50f4f917deafcd526cd5c556f087a4e8"
server_key_version: "encryptionkey@0"
I20260927 22:20:21.830603 13273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "c9e4102d05c04ec5b38db4f879dd4af6"
format_stamp: "Formatted at 2026-09-27 22:20:21 on dist-test-slave-3wc3"
server_key: "fc1a2ed69a046c48ebaab9f0dc729699"
server_key_iv: "50f4f917deafcd526cd5c556f087a4e8"
server_key_version: "encryptionkey@0"
I20260927 22:20:21.835566 13273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:20:21.838967 13288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:21.840200 13273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:20:21.840379 13273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "c9e4102d05c04ec5b38db4f879dd4af6"
format_stamp: "Formatted at 2026-09-27 22:20:21 on dist-test-slave-3wc3"
server_key: "fc1a2ed69a046c48ebaab9f0dc729699"
server_key_iv: "50f4f917deafcd526cd5c556f087a4e8"
server_key_version: "encryptionkey@0"
I20260927 22:20:21.840569 13273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:21.868511 13273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:21.872725 13273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:21.873034 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:21.891846 13273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36249
I20260927 22:20:21.891930 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36249 every 8 connection(s)
I20260927 22:20:21.893195 13273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:21.896900 13340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:21.898909 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13273
I20260927 22:20:21.899106 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:21.899676 24245 external_mini_cluster.cc:1428] Setting key d63004fcb02e4662c18093daf658bcb3
I20260927 22:20:21.910939 13340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6: Bootstrap starting.
I20260927 22:20:21.912542 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:21.901434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:53643 (local address 127.0.0.1:36249)
0927 22:20:21.902275 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:21.902286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:21.902329 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:20:21.903826 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:21.903848 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:21.904234 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:21.904535 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:21.905100 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:21.906405 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:21.908239 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:21.908920 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:21.909036 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:21.910319 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:21.910326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:21.910339 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:21.910875 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:21.910955 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:21.910979 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:21.911151 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:21.911488 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:20:21.911643 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":369,"threads_started":1}
I20260927 22:20:21.914197 13340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:21.915448 13340 log.cc:824] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:21.916242 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:21.918228 13340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6: No bootstrap required, opened a new log
I20260927 22:20:21.927416 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:21.920631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:45897 (local address 127.0.0.1:36249)
0927 22:20:21.920850 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:21.920856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:21.921224 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:20:21.921394 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:21.921398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:21.921458 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:21.921536 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:21.922189 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:21.922930 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:21.924792 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:21.925237 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:21.925324 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:21.926661 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:21.926668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:21.926671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:21.926687 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:21.926725 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:21.926740 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:21.926822 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:21.927030 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:20:21.927071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:20:21.928149 13340 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260927 22:20:21.928570 13340 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:21.928683 13340 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e4102d05c04ec5b38db4f879dd4af6, State: Initialized, Role: FOLLOWER
I20260927 22:20:21.929342 13340 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260927 22:20:21.929590 13340 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:21.929742 13340 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:21.929924 13340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:21.932407 13340 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260927 22:20:21.932928 13340 leader_election.cc:302] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e4102d05c04ec5b38db4f879dd4af6; no voters:
I20260927 22:20:21.933485 13340 leader_election.cc:288] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:21.933584 13350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:21.934333 13350 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [term 1 LEADER]: Becoming Leader. State: Replica: c9e4102d05c04ec5b38db4f879dd4af6, State: Running, Role: LEADER
I20260927 22:20:21.934894 13350 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260927 22:20:21.935635 13340 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:21.938012 13351 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260927 22:20:21.938532 13351 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:21.938114 13352 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e4102d05c04ec5b38db4f879dd4af6. Latest consensus state: current_term: 1 leader_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e4102d05c04ec5b38db4f879dd4af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260927 22:20:21.939337 13357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:21.939371 13352 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:21.944736 13357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:21.955154 13357 catalog_manager.cc:1347] Generated new cluster ID: 8142566b9dd141e4b899d9c8d6413dbc
I20260927 22:20:21.955268 13357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:21.980005 13357 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:21.981827 13357 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:21.990118 13357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e4102d05c04ec5b38db4f879dd4af6: Generated new TSK 0
I20260927 22:20:21.990983 13357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:22.008236 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13273
W20260927 22:20:22.009421 13335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:53643
W20260927 22:20:22.010152 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13273: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:22.010296 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13273
W20260927 22:20:22.027762 13258 connection.cc:535] client connection to 127.0.0.1:36249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36249): Connection reset by peer (error 104)
2026-09-27T22:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:22 dist-test-slave-3wc3 krb5kdc[13371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:22 dist-test-slave-3wc3 krb5kdc[13371](info): set up 2 sockets
Sep 27 22:20:22 dist-test-slave-3wc3 krb5kdc[13371](info): commencing operation
krb5kdc: starting...
W20260927 22:20:23.118376 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:23 dist-test-slave-3wc3 krb5kdc[13371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:20:23.372764 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:23.700904 13384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:23.701239 13384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:23.701407 13384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:23.708849 13384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:23.709009 13384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:23.709122 13384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:23.709236 13384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:23.709316 13384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:23.719169 13384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:23.720463 13384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:23.721804 13384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:23.733814 13389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:23.733771 13390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:23.733852 13392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:23.734743 13384 server_base.cc:1034] running on GCE node
I20260927 22:20:23.735545 13384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:23.736984 13384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:23.738212 13384 hybrid_clock.cc:648] HybridClock initialized: now 1790547623738180 us; error 49 us; skew 500 ppm
I20260927 22:20:23.738562 13384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:23 dist-test-slave-3wc3 krb5kdc[13371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:23.747854 13384 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:20:23.749437 13384 webserver.cc:460] Webserver started at http://127.0.0.1:39475/ using document root <none> and password file <none>
I20260927 22:20:23.750313 13384 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:23.750465 13384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:23.750803 13384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:23.753897 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "5aa699a5a609440c9cd9f0d665d2cb81"
format_stamp: "Formatted at 2026-09-27 22:20:23 on dist-test-slave-3wc3"
server_key: "30ff9f8d2aabc0cda86b7e9b6792bbb7"
server_key_iv: "7f8fd40d88dea5d3949ceb0b527f61c9"
server_key_version: "encryptionkey@0"
I20260927 22:20:23.754774 13384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "5aa699a5a609440c9cd9f0d665d2cb81"
format_stamp: "Formatted at 2026-09-27 22:20:23 on dist-test-slave-3wc3"
server_key: "30ff9f8d2aabc0cda86b7e9b6792bbb7"
server_key_iv: "7f8fd40d88dea5d3949ceb0b527f61c9"
server_key_version: "encryptionkey@0"
I20260927 22:20:23.760110 13384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:20:23.763826 13399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:23.765403 13384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:20:23.765599 13384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "5aa699a5a609440c9cd9f0d665d2cb81"
format_stamp: "Formatted at 2026-09-27 22:20:23 on dist-test-slave-3wc3"
server_key: "30ff9f8d2aabc0cda86b7e9b6792bbb7"
server_key_iv: "7f8fd40d88dea5d3949ceb0b527f61c9"
server_key_version: "encryptionkey@0"
I20260927 22:20:23.765817 13384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:23.771862 13384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:23.776119 13384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:23.776425 13384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:23.795429 13384 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37477
I20260927 22:20:23.795516 13450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37477 every 8 connection(s)
I20260927 22:20:23.796810 13384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:23.797394 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13384
I20260927 22:20:23.797623 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:23.798210 24245 external_mini_cluster.cc:1428] Setting key 1ad5b5a70081eae7824154b14db8919d
I20260927 22:20:23.801411 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:23.805843 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:23.800031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:32971 (local address 127.0.0.1:37477)
0927 22:20:23.800774 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:23.800786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:23.800828 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:20:23.802532 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:23.802554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:23.802874 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:23.803246 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:23.804316 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:23.804322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:23.804335 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:23.804347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:23.804421 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:23.804439 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:23.804674 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:23.804820 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:20:23.804994 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":316,"threads_started":1}
I20260927 22:20:23.809458 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:23.812793 13451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81: Bootstrap starting.
I20260927 22:20:23.816105 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:23.813686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.27:48453 (local address 127.0.0.1:37477)
0927 22:20:23.813855 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:23.813860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:23.814336 (+ 476us) server_negotiation.cc:407] Connection header received
0927 22:20:23.814458 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:23.814463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:23.814539 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:23.814654 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:23.815486 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:23.815492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:23.815496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:23.815524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:23.815575 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:23.815597 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:23.815707 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:23.815790 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 22:20:23.815851 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:20:23.816486 13451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:23.817782 13451 log.cc:824] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:23.820399 13451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81: No bootstrap required, opened a new log
I20260927 22:20:23.827921 13451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260927 22:20:23.828382 13451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:23.828526 13451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa699a5a609440c9cd9f0d665d2cb81, State: Initialized, Role: FOLLOWER
I20260927 22:20:23.829092 13451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260927 22:20:23.829334 13451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:23.829471 13451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:23.829659 13451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:23.831904 13451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260927 22:20:23.832389 13451 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa699a5a609440c9cd9f0d665d2cb81; no voters:
I20260927 22:20:23.832980 13451 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:23.833109 13461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:23.833801 13461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa699a5a609440c9cd9f0d665d2cb81, State: Running, Role: LEADER
I20260927 22:20:23.834395 13461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260927 22:20:23.835130 13451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:23.837550 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa699a5a609440c9cd9f0d665d2cb81. Latest consensus state: current_term: 1 leader_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260927 22:20:23.837989 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:23.837671 13462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa699a5a609440c9cd9f0d665d2cb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260927 22:20:23.838112 13462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:23.840567 13468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:23.845484 13468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:23.854768 13468 catalog_manager.cc:1347] Generated new cluster ID: 0a8fe721732f4a2ab4bfee16b5d4e994
I20260927 22:20:23.854889 13468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:23.866333 13468 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:23.866737 13468 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:23.873581 13468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa699a5a609440c9cd9f0d665d2cb81: Generated new TSK 0
I20260927 22:20:23.874429 13468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:23.887954 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13384
W20260927 22:20:23.889189 13446 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.27:32971
W20260927 22:20:23.889850 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:23.890024 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13384
2026-09-27T22:20:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:23 dist-test-slave-3wc3 krb5kdc[13482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:23 dist-test-slave-3wc3 krb5kdc[13482](info): set up 2 sockets
Sep 27 22:20:23 dist-test-slave-3wc3 krb5kdc[13482](info): commencing operation
krb5kdc: starting...
W20260927 22:20:24.994367 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.014s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:25.249719 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34139
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:34139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:25.573545 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:25.573848 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:25.573945 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:25.581163 13495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:25.581267 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:25.581313 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:25.581403 13495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:25.581470 13495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:25.591300 13495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:34139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:25.592535 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:25.593864 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:25.605599 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:25.605533 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:25.605568 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:25.606804 13495 server_base.cc:1034] running on GCE node
I20260927 22:20:25.607571 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:25.608971 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:25.610203 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790547625610156 us; error 55 us; skew 500 ppm
I20260927 22:20:25.610558 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547625, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:25.613466 13495 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:25.615001 13495 webserver.cc:460] Webserver started at http://127.23.173.126:33863/ using document root <none> and password file <none>
I20260927 22:20:25.615687 13495 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:25.615835 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:25.616132 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:25.619100 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "324766984a7a4762b4834672acd52e63"
format_stamp: "Formatted at 2026-09-27 22:20:25 on dist-test-slave-3wc3"
server_key: "38120f44e3685365096b643eacb9936d"
server_key_iv: "aa99a6f046b118bd5721284fa114b278"
server_key_version: "encryptionkey@0"
I20260927 22:20:25.619858 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "324766984a7a4762b4834672acd52e63"
format_stamp: "Formatted at 2026-09-27 22:20:25 on dist-test-slave-3wc3"
server_key: "38120f44e3685365096b643eacb9936d"
server_key_iv: "aa99a6f046b118bd5721284fa114b278"
server_key_version: "encryptionkey@0"
I20260927 22:20:25.624868 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260927 22:20:25.628259 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:25.629593 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:20:25.629777 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "324766984a7a4762b4834672acd52e63"
format_stamp: "Formatted at 2026-09-27 22:20:25 on dist-test-slave-3wc3"
server_key: "38120f44e3685365096b643eacb9936d"
server_key_iv: "aa99a6f046b118bd5721284fa114b278"
server_key_version: "encryptionkey@0"
I20260927 22:20:25.629992 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:25.654230 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:25.658107 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:25.658375 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:25.677774 13495 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:34139
I20260927 22:20:25.677812 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:34139 every 8 connection(s)
I20260927 22:20:25.679159 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:25.682767 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:25.683789 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13495
I20260927 22:20:25.683981 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:25.684551 24245 external_mini_cluster.cc:1428] Setting key 1238256ec942794f23414e148693b947
I20260927 22:20:25.696601 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: Bootstrap starting.
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:20:25.700280 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:25.701337 13562 log.cc:824] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:25.704594 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: No bootstrap required, opened a new log
I20260927 22:20:25.708365 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:25.686434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46743 (local address 127.23.173.126:34139)
0927 22:20:25.687226 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:25.687240 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:25.687285 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:20:25.688910 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:25.688934 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:25.689327 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:25.689643 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:25.691380 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:25.692625 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:25.694460 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:25.695205 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:25.695324 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:25.700104 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:25.700112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:25.700125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:25.700175 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:25.703881 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:25.704651 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:25.704668 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:25.704676 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:25.704877 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:25.705324 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:25.705332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:25.705335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:25.706011 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:25.706318 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:25.706687 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:20:25.707033 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":375,"threads_started":1}
I20260927 22:20:25.712201 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:25.714052 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } }
I20260927 22:20:25.714399 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:25.714520 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324766984a7a4762b4834672acd52e63, State: Initialized, Role: FOLLOWER
I20260927 22:20:25.715154 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } }
I20260927 22:20:25.715447 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:25.715582 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:25.715772 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:25.718482 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } }
I20260927 22:20:25.718964 13562 leader_election.cc:302] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324766984a7a4762b4834672acd52e63; no voters:
I20260927 22:20:25.719635 13562 leader_election.cc:288] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:25.719748 13572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:25.721727 13572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [term 1 LEADER]: Becoming Leader. State: Replica: 324766984a7a4762b4834672acd52e63, State: Running, Role: LEADER
I20260927 22:20:25.722108 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:25.722378 13572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } }
I20260927 22:20:25.732007 13574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324766984a7a4762b4834672acd52e63. Latest consensus state: current_term: 1 leader_uuid: "324766984a7a4762b4834672acd52e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } } }
I20260927 22:20:25.732427 13574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:25.732991 13573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324766984a7a4762b4834672acd52e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324766984a7a4762b4834672acd52e63" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 34139 } } }
I20260927 22:20:25.733273 13573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:25.733546 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:25.716550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42989 (local address 127.23.173.126:34139)
0927 22:20:25.716753 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:25.716761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:25.717131 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:20:25.717330 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:25.717336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:25.717410 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:25.717524 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:25.719023 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:25.720005 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:25.721498 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:25.722225 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:25.722367 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:25.725526 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:25.725531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:25.725534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:25.725564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:25.728778 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:25.729585 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:25.729599 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:25.729602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:25.729716 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:25.730329 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:25.730334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:25.730336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:25.730822 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:25.732743 (+ 1921us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:25.732938 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:20:25.733141 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 22:20:25.741015 13586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:20:25.741139 13586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:20:25.741298 13587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:25.743590 13587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:25.752280 13587 catalog_manager.cc:1347] Generated new cluster ID: 92213adba8a543c08201ded1e571339c
I20260927 22:20:25.752398 13587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:25.767700 13587 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:25.768949 13587 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:25.776253 13587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 324766984a7a4762b4834672acd52e63: Generated new TSK 0
I20260927 22:20:25.777030 13587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:25.786744 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13495
I20260927 22:20:25.787791 13557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:25.879557 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13495
W20260927 22:20:25.896499 13480 connection.cc:535] client connection to 127.23.173.126:34139 recv error: Network error: failed to read from TLS socket (remote: 127.23.173.126:34139): Connection reset by peer (error 104)
2026-09-27T22:20:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13594](info): set up 2 sockets
Sep 27 22:20:25 dist-test-slave-3wc3 krb5kdc[13594](info): commencing operation
krb5kdc: starting...
W20260927 22:20:26.974417 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:27 dist-test-slave-3wc3 krb5kdc[13594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:27.237597 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41533
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:27.560526 13607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:27.560866 13607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:27.561040 13607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:27.568259 13607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:27.568403 13607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:27.568507 13607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:27.568619 13607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:27.568712 13607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:27.578348 13607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:41533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:27.579630 13607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:27.580998 13607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:27.592658 13612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:27.592763 13613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:27.593345 13607 server_base.cc:1034] running on GCE node
W20260927 22:20:27.593526 13615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:27.594339 13607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:27.595605 13607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:27.596805 13607 hybrid_clock.cc:648] HybridClock initialized: now 1790547627596773 us; error 32 us; skew 500 ppm
I20260927 22:20:27.597129 13607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:27 dist-test-slave-3wc3 krb5kdc[13594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547627, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:27.600140 13607 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:27.601536 13607 webserver.cc:460] Webserver started at http://127.23.173.126:39425/ using document root <none> and password file <none>
I20260927 22:20:27.602281 13607 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:27.602408 13607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:27.602717 13607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:27.605575 13607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "821f7a9ced8f440e9e8ac0aececc7382"
format_stamp: "Formatted at 2026-09-27 22:20:27 on dist-test-slave-3wc3"
server_key: "61dc89e465e2f805e71592354fdacfe7"
server_key_iv: "702de07037816befd87c1e65352058a4"
server_key_version: "encryptionkey@0"
I20260927 22:20:27.606361 13607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "821f7a9ced8f440e9e8ac0aececc7382"
format_stamp: "Formatted at 2026-09-27 22:20:27 on dist-test-slave-3wc3"
server_key: "61dc89e465e2f805e71592354fdacfe7"
server_key_iv: "702de07037816befd87c1e65352058a4"
server_key_version: "encryptionkey@0"
I20260927 22:20:27.611308 13607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:20:27.614758 13622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:27.615978 13607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:27.616151 13607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "821f7a9ced8f440e9e8ac0aececc7382"
format_stamp: "Formatted at 2026-09-27 22:20:27 on dist-test-slave-3wc3"
server_key: "61dc89e465e2f805e71592354fdacfe7"
server_key_iv: "702de07037816befd87c1e65352058a4"
server_key_version: "encryptionkey@0"
I20260927 22:20:27.616345 13607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:27.629251 13607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:27.633356 13607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:27.633643 13607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:27.652477 13607 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:41533
I20260927 22:20:27.652612 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:41533 every 8 connection(s)
I20260927 22:20:27.653863 13607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:27.657516 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:27.660292 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13607
I20260927 22:20:27.660478 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:27.660926 24245 external_mini_cluster.cc:1428] Setting key 4bf6a3ce4fc8d22fcd3fb81f65f0e5cd
I20260927 22:20:27.668588 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382: Bootstrap starting.
I20260927 22:20:27.671772 13674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:27.672899 13674 log.cc:824] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:27.673609 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:27.662678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60627 (local address 127.23.173.126:41533)
0927 22:20:27.663438 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:27.663449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:27.663489 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:20:27.665133 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:27.665155 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:27.665545 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:27.665929 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:27.666495 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:27.667776 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:27.669092 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:27.669746 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:27.669958 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:27.671226 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:27.671233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:27.671247 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:27.671761 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:27.671858 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:27.671907 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:27.672086 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:27.672363 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:20:27.672528 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":329,"threads_started":1}
I20260927 22:20:27.676005 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382: No bootstrap required, opened a new log
I20260927 22:20:27.677632 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:27.684484 13674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } }
I20260927 22:20:27.684870 13674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:27.684969 13674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821f7a9ced8f440e9e8ac0aececc7382, State: Initialized, Role: FOLLOWER
I20260927 22:20:27.685597 13674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } }
I20260927 22:20:27.685853 13674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:27.686065 13674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:27.686254 13674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:27.688501 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:27.682090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58765 (local address 127.23.173.126:41533)
0927 22:20:27.682255 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:27.682280 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:27.682716 (+ 436us) server_negotiation.cc:407] Connection header received
0927 22:20:27.682919 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:27.682924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:27.683000 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:27.683121 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:27.683624 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:27.684396 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:27.685831 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:27.686503 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:27.686643 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:27.687737 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:27.687742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:27.687747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:27.687772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:27.687831 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:27.687847 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:27.687978 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:27.688164 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:20:27.688229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:20:27.689050 13674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } }
I20260927 22:20:27.689558 13674 leader_election.cc:302] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 821f7a9ced8f440e9e8ac0aececc7382; no voters:
I20260927 22:20:27.690233 13674 leader_election.cc:288] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:27.690346 13684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:27.691066 13684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [term 1 LEADER]: Becoming Leader. State: Replica: 821f7a9ced8f440e9e8ac0aececc7382, State: Running, Role: LEADER
I20260927 22:20:27.691627 13684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } }
I20260927 22:20:27.692615 13674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:27.695715 13686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 821f7a9ced8f440e9e8ac0aececc7382. Latest consensus state: current_term: 1 leader_uuid: "821f7a9ced8f440e9e8ac0aececc7382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } } }
I20260927 22:20:27.695792 13685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "821f7a9ced8f440e9e8ac0aececc7382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821f7a9ced8f440e9e8ac0aececc7382" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 41533 } } }
I20260927 22:20:27.696085 13686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:27.697662 13691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:27.697669 13685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:27.703539 13691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:27.713452 13691 catalog_manager.cc:1347] Generated new cluster ID: cbcd91685ea144898be5dfe94251f605
I20260927 22:20:27.713580 13691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:27.776446 13691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:27.778378 13691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:27.789778 13691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 821f7a9ced8f440e9e8ac0aececc7382: Generated new TSK 0
I20260927 22:20:27.790898 13691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:27.807293 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13607
W20260927 22:20:27.808665 13669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60627
W20260927 22:20:27.809451 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:27.809600 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13607
W20260927 22:20:27.827013 13592 connection.cc:535] client connection to 127.23.173.126:41533 recv error: Network error: failed to read from TLS socket (remote: 127.23.173.126:41533): Connection reset by peer (error 104)
2026-09-27T22:20:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:27 dist-test-slave-3wc3 krb5kdc[13705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:27 dist-test-slave-3wc3 krb5kdc[13705](info): set up 2 sockets
Sep 27 22:20:27 dist-test-slave-3wc3 krb5kdc[13705](info): commencing operation
krb5kdc: starting...
W20260927 22:20:28.922421 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:29 dist-test-slave-3wc3 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:29.173887 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43909
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:29.495065 13718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:29.495395 13718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:29.495571 13718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:29.502652 13718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:29.502791 13718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:29.502888 13718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:29.502996 13718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:29.503085 13718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:29.512562 13718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:43909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:29.513813 13718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:29.515151 13718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:29.526988 13723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:29.527061 13724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:29.527104 13726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:29.528200 13718 server_base.cc:1034] running on GCE node
I20260927 22:20:29.528959 13718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:29.530370 13718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:29.531600 13718 hybrid_clock.cc:648] HybridClock initialized: now 1790547629531566 us; error 44 us; skew 500 ppm
I20260927 22:20:29.531937 13718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:29 dist-test-slave-3wc3 krb5kdc[13705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:29.538623 13718 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:29.540097 13718 webserver.cc:460] Webserver started at http://127.23.173.126:38113/ using document root <none> and password file <none>
I20260927 22:20:29.540756 13718 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:29.540899 13718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:29.541204 13718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:29.544207 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "0891ba890c69446aa98f3dd073653013"
format_stamp: "Formatted at 2026-09-27 22:20:29 on dist-test-slave-3wc3"
server_key: "5b47fede4f8f5d1fc488504d40f6f6cb"
server_key_iv: "1973712d8e4116b9518dbe752bb11f6d"
server_key_version: "encryptionkey@0"
I20260927 22:20:29.544950 13718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "0891ba890c69446aa98f3dd073653013"
format_stamp: "Formatted at 2026-09-27 22:20:29 on dist-test-slave-3wc3"
server_key: "5b47fede4f8f5d1fc488504d40f6f6cb"
server_key_iv: "1973712d8e4116b9518dbe752bb11f6d"
server_key_version: "encryptionkey@0"
I20260927 22:20:29.549923 13718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:20:29.553388 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:29.554769 13718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:20:29.554955 13718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "0891ba890c69446aa98f3dd073653013"
format_stamp: "Formatted at 2026-09-27 22:20:29 on dist-test-slave-3wc3"
server_key: "5b47fede4f8f5d1fc488504d40f6f6cb"
server_key_iv: "1973712d8e4116b9518dbe752bb11f6d"
server_key_version: "encryptionkey@0"
I20260927 22:20:29.555186 13718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:29.560889 13718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:29.565229 13718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:29.565521 13718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:29.582746 13718 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:43909
I20260927 22:20:29.582768 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:43909 every 8 connection(s)
I20260927 22:20:29.584097 13718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:29.587658 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:29.588111 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13718
I20260927 22:20:29.588275 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:29.588734 24245 external_mini_cluster.cc:1428] Setting key 716dd4f465a57735eea27a676adcdce1
I20260927 22:20:29.596794 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:29.590452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52649 (local address 127.23.173.126:43909)
0927 22:20:29.591269 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:29.591280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:29.591319 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:20:29.593466 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:29.593488 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:29.593809 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:29.594258 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:29.595297 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:29.595303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:29.595316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:29.595329 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:29.595382 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:29.595400 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:29.595629 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:29.595758 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 22:20:29.595945 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":350,"threads_started":1}
I20260927 22:20:29.600391 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:29.600394 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013: Bootstrap starting.
I20260927 22:20:29.603456 13785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:29.604503 13785 log.cc:824] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:29.607276 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:29.604503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40405 (local address 127.23.173.126:43909)
0927 22:20:29.604704 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:29.604711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:29.605213 (+ 502us) server_negotiation.cc:407] Connection header received
0927 22:20:29.605483 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:29.605490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:29.605545 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:29.605621 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:29.606737 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:29.606741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:29.606744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:29.606758 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:29.606793 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:29.606807 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:29.606895 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:29.607032 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:20:29.607072 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:20:29.607990 13785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013: No bootstrap required, opened a new log
I20260927 22:20:29.615641 13785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } }
I20260927 22:20:29.615942 13785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:29.616003 13785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0891ba890c69446aa98f3dd073653013, State: Initialized, Role: FOLLOWER
I20260927 22:20:29.616542 13785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } }
I20260927 22:20:29.616743 13785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:29.616839 13785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:29.616954 13785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:29.619105 13785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } }
I20260927 22:20:29.619705 13785 leader_election.cc:302] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0891ba890c69446aa98f3dd073653013; no voters:
I20260927 22:20:29.620313 13785 leader_election.cc:288] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:29.620455 13795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:29.621150 13795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [term 1 LEADER]: Becoming Leader. State: Replica: 0891ba890c69446aa98f3dd073653013, State: Running, Role: LEADER
I20260927 22:20:29.621657 13795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } }
I20260927 22:20:29.622437 13785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:29.624361 13796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0891ba890c69446aa98f3dd073653013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } } }
I20260927 22:20:29.624742 13796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:29.626202 13800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:29.626195 13797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0891ba890c69446aa98f3dd073653013. Latest consensus state: current_term: 1 leader_uuid: "0891ba890c69446aa98f3dd073653013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0891ba890c69446aa98f3dd073653013" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 43909 } } }
I20260927 22:20:29.626500 13797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:29.631232 13800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:29.640280 13800 catalog_manager.cc:1347] Generated new cluster ID: a57bd3d31e8f484c9df99c5ceaf14b56
I20260927 22:20:29.640390 13800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:29.648030 13800 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:29.648397 13800 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:29.658643 13800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0891ba890c69446aa98f3dd073653013: Generated new TSK 0
I20260927 22:20:29.659555 13800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:29.664798 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13718
W20260927 22:20:29.666280 13780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52649
W20260927 22:20:29.667001 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:29.667153 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13718
2026-09-27T22:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:29 dist-test-slave-3wc3 krb5kdc[13816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:29 dist-test-slave-3wc3 krb5kdc[13816](info): set up 2 sockets
Sep 27 22:20:29 dist-test-slave-3wc3 krb5kdc[13816](info): commencing operation
krb5kdc: starting...
W20260927 22:20:30.770337 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:30 dist-test-slave-3wc3 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:31.021196 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:46143
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:46143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:31.346635 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:31.346915 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:31.347112 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:31.354213 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:31.354350 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:31.354434 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:31.354552 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:31.354632 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:31.364208 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:46143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:46143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:31.365470 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:31.366761 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:31.378335 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:31.378458 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:31.378372 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:31.379086 13829 server_base.cc:1034] running on GCE node
I20260927 22:20:31.379839 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:31.381239 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:31.382460 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790547631382412 us; error 42 us; skew 500 ppm
I20260927 22:20:31.382807 13829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:31 dist-test-slave-3wc3 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547631, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:31.385818 13829 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:31.387358 13829 webserver.cc:460] Webserver started at http://127.23.173.126:45339/ using document root <none> and password file <none>
I20260927 22:20:31.388029 13829 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:31.388168 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:31.388465 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:31.391392 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "65c1fc6b81ef4953bc5d0e4bb3551534"
format_stamp: "Formatted at 2026-09-27 22:20:31 on dist-test-slave-3wc3"
server_key: "5cef221d94d2beee0babafeb78845e51"
server_key_iv: "6791ba888e7aa510bb73b6ef87deafdc"
server_key_version: "encryptionkey@0"
I20260927 22:20:31.392124 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "65c1fc6b81ef4953bc5d0e4bb3551534"
format_stamp: "Formatted at 2026-09-27 22:20:31 on dist-test-slave-3wc3"
server_key: "5cef221d94d2beee0babafeb78845e51"
server_key_iv: "6791ba888e7aa510bb73b6ef87deafdc"
server_key_version: "encryptionkey@0"
I20260927 22:20:31.396981 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:20:31.400382 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:31.401635 13829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:31.401823 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "65c1fc6b81ef4953bc5d0e4bb3551534"
format_stamp: "Formatted at 2026-09-27 22:20:31 on dist-test-slave-3wc3"
server_key: "5cef221d94d2beee0babafeb78845e51"
server_key_iv: "6791ba888e7aa510bb73b6ef87deafdc"
server_key_version: "encryptionkey@0"
I20260927 22:20:31.402035 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:31.417137 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:31.421586 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:31.421905 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:31.443348 13829 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:46143
I20260927 22:20:31.443372 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:46143 every 8 connection(s)
I20260927 22:20:31.444845 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:31.445834 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13829
I20260927 22:20:31.446038 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:31.446480 24245 external_mini_cluster.cc:1428] Setting key 76c50837bef894c4218185c152ae747b
I20260927 22:20:31.449362 13896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:20:31 dist-test-slave-3wc3 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:20:31.462342 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534: Bootstrap starting.
I20260927 22:20:31.466696 13896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:31.467948 13896 log.cc:824] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:31.471462 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534: No bootstrap required, opened a new log
I20260927 22:20:31.471498 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:31.448356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54485 (local address 127.23.173.126:46143)
0927 22:20:31.449132 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:31.449150 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:31.449209 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:20:31.451514 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:31.451543 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:31.452042 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:31.452491 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:31.453790 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:31.455940 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:31.457632 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:31.458722 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:31.458918 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:31.462951 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:31.462963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:31.462984 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:31.463035 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:31.466948 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:31.467626 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:31.467634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:31.467644 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:31.467860 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:31.468266 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:31.468271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:31.468274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:31.468965 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:31.469193 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:31.469749 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 22:20:31.470215 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":310,"threads_started":1}
I20260927 22:20:31.475267 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:31.479857 13896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } }
I20260927 22:20:31.480369 13896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:31.480491 13896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c1fc6b81ef4953bc5d0e4bb3551534, State: Initialized, Role: FOLLOWER
I20260927 22:20:31.481210 13896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } }
I20260927 22:20:31.481513 13896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:31.481659 13896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:31.481869 13896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:31.484926 13896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } }
I20260927 22:20:31.485527 13896 leader_election.cc:302] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c1fc6b81ef4953bc5d0e4bb3551534; no voters:
I20260927 22:20:31.486248 13896 leader_election.cc:288] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:31.486371 13906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:31.489169 13896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:31.490253 13906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [term 1 LEADER]: Becoming Leader. State: Replica: 65c1fc6b81ef4953bc5d0e4bb3551534, State: Running, Role: LEADER
I20260927 22:20:31.491004 13906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } }
I20260927 22:20:31.494771 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c1fc6b81ef4953bc5d0e4bb3551534. Latest consensus state: current_term: 1 leader_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } } }
I20260927 22:20:31.495229 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:31.495674 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:31.479549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49877 (local address 127.23.173.126:46143)
0927 22:20:31.479779 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:31.479789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:31.480341 (+ 552us) server_negotiation.cc:407] Connection header received
0927 22:20:31.480530 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:31.480536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:31.480642 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:31.480739 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:31.482768 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:31.483587 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:31.485350 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:31.485878 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:31.486058 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:31.488708 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:31.488715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:31.488719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:31.488763 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:31.492001 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:31.492637 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:31.492642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:31.492646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:31.492795 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:31.493158 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:31.493165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:31.493169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:31.493583 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:31.494880 (+ 1297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:31.495041 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:20:31.495271 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:31.495867 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c1fc6b81ef4953bc5d0e4bb3551534" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 46143 } } }
I20260927 22:20:31.496752 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:31.499042 13911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:31.506135 13911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:31.516971 13911 catalog_manager.cc:1347] Generated new cluster ID: ea2e2fe3b6094f1ba48be2fe03051bdc
I20260927 22:20:31.517077 13911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:31.527837 13911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:31.529778 13911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:31.542750 13911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c1fc6b81ef4953bc5d0e4bb3551534: Generated new TSK 0
I20260927 22:20:31.543814 13911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:31.549993 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13829
I20260927 22:20:31.551294 13891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:31.642997 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13829
2026-09-27T22:20:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:31 dist-test-slave-3wc3 krb5kdc[13929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:31 dist-test-slave-3wc3 krb5kdc[13929](info): set up 2 sockets
Sep 27 22:20:31 dist-test-slave-3wc3 krb5kdc[13929](info): commencing operation
krb5kdc: starting...
W20260927 22:20:32.754519 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:32 dist-test-slave-3wc3 krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:33.008895 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42529
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:33.333176 13942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:33.333521 13942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:33.333688 13942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:33.341012 13942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:33.341167 13942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:33.341280 13942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:33.341394 13942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:33.341480 13942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:33.351315 13942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:33.352645 13942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:33.354027 13942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:33.365656 13947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:33.365685 13948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:33.365614 13950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:33.366577 13942 server_base.cc:1034] running on GCE node
I20260927 22:20:33.367345 13942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:33.368714 13942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:33.369910 13942 hybrid_clock.cc:648] HybridClock initialized: now 1790547633369862 us; error 45 us; skew 500 ppm
I20260927 22:20:33.370306 13942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:33 dist-test-slave-3wc3 krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547633, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:33.373489 13942 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:33.375116 13942 webserver.cc:460] Webserver started at http://127.23.173.126:33429/ using document root <none> and password file <none>
I20260927 22:20:33.375797 13942 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:33.375942 13942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:33.376243 13942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:33.379251 13942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "fb53607204c2417a81175e83f72796b4"
format_stamp: "Formatted at 2026-09-27 22:20:33 on dist-test-slave-3wc3"
server_key: "6f3c2a70c95f357799466d2972117bfa"
server_key_iv: "db7046de49e6505bc355ba1cd8c3788a"
server_key_version: "encryptionkey@0"
I20260927 22:20:33.380005 13942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "fb53607204c2417a81175e83f72796b4"
format_stamp: "Formatted at 2026-09-27 22:20:33 on dist-test-slave-3wc3"
server_key: "6f3c2a70c95f357799466d2972117bfa"
server_key_iv: "db7046de49e6505bc355ba1cd8c3788a"
server_key_version: "encryptionkey@0"
I20260927 22:20:33.385103 13942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:20:33.388523 13957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:33.389895 13942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:20:33.390125 13942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "fb53607204c2417a81175e83f72796b4"
format_stamp: "Formatted at 2026-09-27 22:20:33 on dist-test-slave-3wc3"
server_key: "6f3c2a70c95f357799466d2972117bfa"
server_key_iv: "db7046de49e6505bc355ba1cd8c3788a"
server_key_version: "encryptionkey@0"
I20260927 22:20:33.390291 13942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:33.401997 13942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:33.406276 13942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:33.406561 13942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:33.425071 13942 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42529
I20260927 22:20:33.425098 14008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42529 every 8 connection(s)
I20260927 22:20:33.426448 13942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:33.430229 14009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:33.432436 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 13942
I20260927 22:20:33.432611 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:33.433037 24245 external_mini_cluster.cc:1428] Setting key 4516005ae3751f5db36c4703583b51d0
I20260927 22:20:33.443675 14009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: Bootstrap starting.
I20260927 22:20:33.447530 14009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:33.448325 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:33.434868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59679 (local address 127.23.173.126:42529)
0927 22:20:33.435704 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:33.435724 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:33.435776 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:20:33.438115 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:33.438141 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:33.438653 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:33.439088 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:33.439609 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:33.441800 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:33.443109 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:33.444163 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:33.444344 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:33.445466 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:33.445482 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:33.445504 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:33.446259 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:33.446357 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:33.446390 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:33.446515 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:33.446710 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:20:33.446962 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":332,"threads_started":1}
I20260927 22:20:33.448884 14009 log.cc:824] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:33.452157 14009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: No bootstrap required, opened a new log
I20260927 22:20:33.452198 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:33.460273 14009 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } }
I20260927 22:20:33.460727 14009 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:33.460856 14009 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb53607204c2417a81175e83f72796b4, State: Initialized, Role: FOLLOWER
I20260927 22:20:33.461499 14009 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } }
I20260927 22:20:33.461764 14009 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:33.461932 14009 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:33.462157 14009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:33.462605 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:33.456492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44659 (local address 127.23.173.126:42529)
0927 22:20:33.456676 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:33.456684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:33.457045 (+ 361us) server_negotiation.cc:407] Connection header received
0927 22:20:33.457265 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:33.457269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:33.457336 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:33.457454 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:33.457980 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:33.458720 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:33.460042 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:33.460473 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:33.460611 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:33.461771 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:33.461775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:33.461778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:33.461795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:33.461833 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:33.461849 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:33.461983 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:33.462173 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:20:33.462274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:20:33.464933 14009 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } }
I20260927 22:20:33.465437 14009 leader_election.cc:302] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb53607204c2417a81175e83f72796b4; no voters:
I20260927 22:20:33.466193 14009 leader_election.cc:288] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:33.466331 14019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:33.467706 14019 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [term 1 LEADER]: Becoming Leader. State: Replica: fb53607204c2417a81175e83f72796b4, State: Running, Role: LEADER
I20260927 22:20:33.468222 14019 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } }
I20260927 22:20:33.468590 14009 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:33.471505 14021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb53607204c2417a81175e83f72796b4. Latest consensus state: current_term: 1 leader_uuid: "fb53607204c2417a81175e83f72796b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } } }
I20260927 22:20:33.471861 14021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:33.472285 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb53607204c2417a81175e83f72796b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb53607204c2417a81175e83f72796b4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42529 } } }
I20260927 22:20:33.472674 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4 [sys.catalog]: This master's current role is: LEADER
W20260927 22:20:33.480189 14035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:20:33.480360 14035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:20:33.480504 14027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:33.482125 14027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:33.490310 14027 catalog_manager.cc:1347] Generated new cluster ID: 0482e474c7394b909f63517855ca9f29
I20260927 22:20:33.490425 14027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:33.514279 14027 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:33.515554 14027 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:33.524439 14027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb53607204c2417a81175e83f72796b4: Generated new TSK 0
I20260927 22:20:33.525285 14027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:33.530507 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13942
W20260927 22:20:33.531545 14004 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59679
W20260927 22:20:33.532207 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 13942: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:33.532347 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 13942
2026-09-27T22:20:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:33 dist-test-slave-3wc3 krb5kdc[14040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:33 dist-test-slave-3wc3 krb5kdc[14040](info): set up 2 sockets
Sep 27 22:20:33 dist-test-slave-3wc3 krb5kdc[14040](info): commencing operation
krb5kdc: starting...
W20260927 22:20:34.630512 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:34 dist-test-slave-3wc3 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:34.896728 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42621
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:35.229079 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:35.229449 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:35.229631 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:35.236889 14053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:35.237059 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:35.237172 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:35.237267 14053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:35.237340 14053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:35.247278 14053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:42621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:35.248672 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:35.250061 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:35.262585 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:35.262279 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:35.262625 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:35.264175 14053 server_base.cc:1034] running on GCE node
I20260927 22:20:35.264933 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:35.266450 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:35.267663 14053 hybrid_clock.cc:648] HybridClock initialized: now 1790547635267605 us; error 43 us; skew 500 ppm
I20260927 22:20:35.268026 14053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:35 dist-test-slave-3wc3 krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:35.276809 14053 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:35.278568 14053 webserver.cc:460] Webserver started at http://127.23.173.126:38683/ using document root <none> and password file <none>
I20260927 22:20:35.279277 14053 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:35.279417 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:35.279743 14053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:35.282846 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "6508fd2122914782a2395a833d752cd0"
format_stamp: "Formatted at 2026-09-27 22:20:35 on dist-test-slave-3wc3"
server_key: "9f597513ecc114e824a66aaeddb319dd"
server_key_iv: "6340f04978189b487f7fed344ab2c220"
server_key_version: "encryptionkey@0"
I20260927 22:20:35.283633 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "6508fd2122914782a2395a833d752cd0"
format_stamp: "Formatted at 2026-09-27 22:20:35 on dist-test-slave-3wc3"
server_key: "9f597513ecc114e824a66aaeddb319dd"
server_key_iv: "6340f04978189b487f7fed344ab2c220"
server_key_version: "encryptionkey@0"
I20260927 22:20:35.288643 14053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:20:35.292040 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:35.293322 14053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:35.293510 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "6508fd2122914782a2395a833d752cd0"
format_stamp: "Formatted at 2026-09-27 22:20:35 on dist-test-slave-3wc3"
server_key: "9f597513ecc114e824a66aaeddb319dd"
server_key_iv: "6340f04978189b487f7fed344ab2c220"
server_key_version: "encryptionkey@0"
I20260927 22:20:35.293712 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:35.299351 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:35.303179 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:35.303478 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:35.322041 14053 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:42621
I20260927 22:20:35.322077 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:42621 every 8 connection(s)
I20260927 22:20:35.323462 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:35.327131 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:35.331694 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14053
I20260927 22:20:35.331867 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:35.332432 24245 external_mini_cluster.cc:1428] Setting key b5735f39c6eb3ec20e8c4084f79933f7
I20260927 22:20:35.338441 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0: Bootstrap starting.
I20260927 22:20:35.340269 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:35.334238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43201 (local address 127.23.173.126:42621)
0927 22:20:35.335060 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:35.335072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:35.335113 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:20:35.336944 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:35.336965 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:35.337290 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:35.337604 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:35.338676 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:35.338682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:35.338695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:35.338713 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:35.338776 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:35.338802 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:35.339078 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:35.339213 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 22:20:35.339385 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":356,"threads_started":1}
I20260927 22:20:35.341579 14120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:35.342762 14120 log.cc:824] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:35.344005 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:35.345592 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0: No bootstrap required, opened a new log
I20260927 22:20:35.350698 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:35.348221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59329 (local address 127.23.173.126:42621)
0927 22:20:35.348395 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:35.348403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:35.348769 (+ 366us) server_negotiation.cc:407] Connection header received
0927 22:20:35.348901 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:35.348905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:35.348968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:35.349082 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:35.350044 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:35.350049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:35.350054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:20:35.350077 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:35.350127 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:20:35.350148 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:35.350271 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:35.350329 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:20:35.350379 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:35.354375 14120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } }
I20260927 22:20:35.354734 14120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:20:35.354875 14120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6508fd2122914782a2395a833d752cd0, State: Initialized, Role: FOLLOWER
I20260927 22:20:35.355428 14120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } }
I20260927 22:20:35.355679 14120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:20:35.355835 14120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:20:35.356060 14120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:35.358378 14120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } }
I20260927 22:20:35.358914 14120 leader_election.cc:302] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6508fd2122914782a2395a833d752cd0; no voters:
I20260927 22:20:35.359573 14120 leader_election.cc:288] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:20:35.359686 14130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:35.360469 14130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 6508fd2122914782a2395a833d752cd0, State: Running, Role: LEADER
I20260927 22:20:35.361012 14130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } }
I20260927 22:20:35.361771 14120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:35.364027 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6508fd2122914782a2395a833d752cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } } }
I20260927 22:20:35.364172 14132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6508fd2122914782a2395a833d752cd0. Latest consensus state: current_term: 1 leader_uuid: "6508fd2122914782a2395a833d752cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6508fd2122914782a2395a833d752cd0" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 42621 } } }
I20260927 22:20:35.364562 14132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:35.364566 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:35.365552 14138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:35.371986 14138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:35.381261 14138 catalog_manager.cc:1347] Generated new cluster ID: ee6b3e149f9d489dab98013e7144db32
I20260927 22:20:35.381415 14138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:35.403260 14138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:35.403806 14138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:35.422983 14138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6508fd2122914782a2395a833d752cd0: Generated new TSK 0
I20260927 22:20:35.423957 14138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:35.438187 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14053
W20260927 22:20:35.439262 14115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43201
W20260927 22:20:35.440025 24245 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:20:35.440191 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14053
2026-09-27T22:20:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1906 ms)
[----------] 18 tests from AuthTokenIssuingTest (34981 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:35 dist-test-slave-3wc3 krb5kdc[14151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:35 dist-test-slave-3wc3 krb5kdc[14151](info): set up 2 sockets
Sep 27 22:20:35 dist-test-slave-3wc3 krb5kdc[14151](info): commencing operation
krb5kdc: starting...
W20260927 22:20:36.550585 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:36 dist-test-slave-3wc3 krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:36.823369 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37857
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:37.155303 14164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:37.155629 14164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:37.155701 14164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:37.155786 14164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:37.163149 14164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:37.163256 14164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:37.163313 14164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:37.163357 14164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:37.163419 14164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:37.173727 14164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:37.175375 14164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:37.176651 14164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:37.189373 14169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:37.189431 14172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:37.189321 14170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:37.190444 14164 server_base.cc:1034] running on GCE node
I20260927 22:20:37.191244 14164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:37.192662 14164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:37.193917 14164 hybrid_clock.cc:648] HybridClock initialized: now 1790547637193850 us; error 39 us; skew 500 ppm
I20260927 22:20:37.194294 14164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:37.197491 14164 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:37.199074 14164 webserver.cc:460] Webserver started at http://127.23.173.126:38981/ using document root <none> and password file <none>
I20260927 22:20:37.199815 14164 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:37.199949 14164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:37.200280 14164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:37.203399 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "89c0b27b48824208adffc5fccd2c60d4"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "17c66da2e1703aed8682e011081b043b"
server_key_iv: "649d25a1e8491063021d4ac4d7c71f0b"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.204239 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "89c0b27b48824208adffc5fccd2c60d4"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "17c66da2e1703aed8682e011081b043b"
server_key_iv: "649d25a1e8491063021d4ac4d7c71f0b"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.209618 14164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:20:37.213146 14179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:37.214488 14164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:20:37.214677 14164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "89c0b27b48824208adffc5fccd2c60d4"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "17c66da2e1703aed8682e011081b043b"
server_key_iv: "649d25a1e8491063021d4ac4d7c71f0b"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.214900 14164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:37.232944 14164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:37.236907 14164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:37.237188 14164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:37.255640 14164 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:37857
I20260927 22:20:37.255661 14230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:37857 every 8 connection(s)
I20260927 22:20:37.256980 14164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:37.259018 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14164
I20260927 22:20:37.259251 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:37.260073 24245 external_mini_cluster.cc:1428] Setting key 3dec4788cb5a10c7aca8ca3b22312e11
I20260927 22:20:37.261539 14231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:37.271526 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.125@KRBTEST.COM" created.
I20260927 22:20:37.295441 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.274849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.23.173.126:37857)
0927 22:20:37.276551 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:37.276561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:37.276609 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:20:37.278323 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:37.278334 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:37.278669 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:37.278883 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:37.280252 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.282187 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.283584 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.284573 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.284766 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.288247 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:37.288260 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:37.288280 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:37.288327 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:37.291330 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.292153 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.292159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.292162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.292310 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.292758 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.292761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.292763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.293234 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:37.293390 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:37.294134 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 22:20:37.294383 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1493,"spinlock_wait_cycles":34560,"thread_start_us":1286,"threads_started":1}
I20260927 22:20:37.295485 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.274848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37857 (local address 127.0.0.1:47258)
0927 22:20:37.275913 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.276064 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:37.277897 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:37.278938 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:37.278949 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:37.279685 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:37.280033 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.280051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.282349 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.282354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.283429 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.283441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.284793 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.284797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.284917 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.285724 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:37.285747 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:37.288009 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:37.291471 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.291487 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.291500 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.292016 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.292431 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.292438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.292444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.292651 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.293387 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:37.293398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:37.293680 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 22:20:37.294000 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 22:20:37.294599 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":50,"thread_start_us":419,"threads_started":1}
I20260927 22:20:37.298449 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:37.299597 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.299086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35704)
0927 22:20:37.299315 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.299440 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 22:20:37.299829 14182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
I20260927 22:20:37.301419 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.301024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35710)
0927 22:20:37.301186 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.301270 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:20:37.301755 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:46213: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
I20260927 22:20:37.323441 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:46213
--webserver_interface=127.23.173.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:20:37.351130 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } attempt: 1
I20260927 22:20:37.352345 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.351903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35726)
0927 22:20:37.352099 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.352212 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:20:37.353368 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.353074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35730)
0927 22:20:37.353191 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.353256 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 22:20:37.353612 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:46213: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
I20260927 22:20:37.421864 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } attempt: 2
I20260927 22:20:37.423094 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.422624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35736)
0927 22:20:37.422859 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.422956 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:20:37.424160 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.423831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35752)
0927 22:20:37.423971 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.424039 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:20:37.424424 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:46213: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
I20260927 22:20:37.515669 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } attempt: 3
I20260927 22:20:37.516913 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.516478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35764)
0927 22:20:37.516678 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.516784 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:20:37.517930 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.517623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35772)
0927 22:20:37.517752 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.517814 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:20:37.518251 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:46213: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
W20260927 22:20:37.651670 14238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:37.652057 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:37.652189 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:37.652333 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:37.659415 14238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:37.659533 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:37.659639 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:37.659734 14238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:37.659826 14238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:37.661471 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } attempt: 4
I20260927 22:20:37.662683 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.662158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35778)
0927 22:20:37.662468 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.662554 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 22:20:37.663671 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.663305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35794)
0927 22:20:37.663456 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.663535 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:20:37.664009 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:46213: Network error: Client connection negotiation failed: client connection to 127.23.173.125:46213: connect: Connection refused (error 111)
I20260927 22:20:37.670385 14238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:37.671613 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:37.672925 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:37.684578 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:37.684612 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:37.685073 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:37.685483 14238 server_base.cc:1034] running on GCE node
I20260927 22:20:37.686254 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:37.687520 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:37.688746 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790547637688683 us; error 33 us; skew 500 ppm
I20260927 22:20:37.689076 14238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:37.692173 14238 init.cc:377] Logged in from keytab as kudu/127.23.173.125@KRBTEST.COM (short username kudu)
I20260927 22:20:37.693559 14238 webserver.cc:460] Webserver started at http://127.23.173.125:45635/ using document root <none> and password file <none>
I20260927 22:20:37.694288 14238 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:37.694375 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:37.694629 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:37.697448 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data/instance:
uuid: "d2cf883504a34b81b76189b9620ac1ff"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "a3ff1e1a21eb2b9d8e233c9d4fc11170"
server_key_iv: "abe9a84ac6d5453346e2d131c7e4f7ed"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.698269 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal/instance:
uuid: "d2cf883504a34b81b76189b9620ac1ff"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "a3ff1e1a21eb2b9d8e233c9d4fc11170"
server_key_iv: "abe9a84ac6d5453346e2d131c7e4f7ed"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.703100 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:20:37.706373 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:37.707602 14238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:20:37.707787 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "d2cf883504a34b81b76189b9620ac1ff"
format_stamp: "Formatted at 2026-09-27 22:20:37 on dist-test-slave-3wc3"
server_key: "a3ff1e1a21eb2b9d8e233c9d4fc11170"
server_key_iv: "abe9a84ac6d5453346e2d131c7e4f7ed"
server_key_version: "encryptionkey@0"
I20260927 22:20:37.707994 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:37.730223 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:37.734143 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:37.734406 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:37.752761 14238 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.125:46213
I20260927 22:20:37.752765 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.125:46213 every 8 connection(s)
I20260927 22:20:37.754197 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
I20260927 22:20:37.757903 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:37.758217 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14238
I20260927 22:20:37.758399 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-1/wal/instance
I20260927 22:20:37.758926 24245 external_mini_cluster.cc:1428] Setting key 89d534300bc101b7a40916b765eb3b5a
I20260927 22:20:37.767920 14305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.124@KRBTEST.COM" created.
I20260927 22:20:37.789436 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.771291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.23.173.126:37857)
0927 22:20:37.771475 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:37.771480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:37.772754 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 22:20:37.774718 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:37.774724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:37.774816 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:37.774929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:37.776387 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.777065 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.778633 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.779048 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.779183 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.783333 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:37.783338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:37.783341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:37.783373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:37.786211 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.786917 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.786921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.786923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.787036 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.787451 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.787457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.787460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.787772 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:37.787883 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:37.788746 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 22:20:37.788988 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:20:37.790140 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.771849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37857 (local address 127.0.0.1:47272)
0927 22:20:37.772566 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.772602 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:37.774429 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:37.775164 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:37.775172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:37.775673 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:37.776214 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.776230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.777234 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.777238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.778481 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.778493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.779295 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.779304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.779577 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.780251 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:37.780276 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:37.783101 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:37.786330 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.786336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.786348 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.786807 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.787133 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.787137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.787140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.787302 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.787881 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:37.787886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:37.788313 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 22:20:37.788597 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 22:20:37.788987 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":299,"threads_started":1}
I20260927 22:20:37.791996 14305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:37.807251 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.792740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35802)
0927 22:20:37.793021 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.793042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:37.793148 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:37.794815 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:37.794826 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:37.795323 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:37.795531 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.795545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.797029 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.797035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.797913 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.797925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.798780 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.798786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.798982 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.799557 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:37.799574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:37.801591 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:37.804772 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.804778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.804785 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.805140 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.805505 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.805509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.805513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.805656 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.806293 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:37.806298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:37.806438 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 22:20:37.806579 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:20:37.806845 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 22:20:37.807338 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.793030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.23.173.125:46213)
0927 22:20:37.794412 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:37.794418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:37.794444 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:20:37.794484 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:37.794490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:37.794655 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:37.794767 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:37.795766 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.796878 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.798103 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.798638 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.798775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.801803 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:37.801814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:37.801826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:37.801858 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:37.804608 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.805269 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.805277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.805280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.805392 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.805756 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.805764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.805767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.806161 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:37.806375 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:37.806716 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 22:20:37.806952 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":1215,"thread_start_us":1053,"threads_started":1}
I20260927 22:20:37.809595 14305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:37.810909 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.810440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54884)
0927 22:20:37.810631 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.810751 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 22:20:37.811230 14257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:37.812935 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.812436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54886)
0927 22:20:37.812613 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.812751 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 22:20:37.813247 14305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
I20260927 22:20:37.821501 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:39567
--webserver_interface=127.23.173.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:20:37.862566 14305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 1
I20260927 22:20:37.863787 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.863322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54900)
0927 22:20:37.863515 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.863633 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:20:37.864885 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.864581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54912)
0927 22:20:37.864723 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.864791 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:20:37.865207 14305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:37.933447 14305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 2
I20260927 22:20:37.934659 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.934201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54920)
0927 22:20:37.934396 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.934490 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:20:37.935796 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.935434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54924)
0927 22:20:37.935603 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.935675 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:20:37.936058 14305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:37.963325 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } attempt: 5
Sep 27 22:20:37 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:37.977003 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.964346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.23.173.125:46213)
0927 22:20:37.964559 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:37.964564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:37.964596 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:20:37.964691 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:37.964695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:37.964756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:37.964866 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:37.965715 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.966476 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.967622 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.968097 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.968228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.971664 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:37.971669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:37.971672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:37.971703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:37.974425 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.975108 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.975114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.975119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.975246 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:37.975623 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:37.975629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:37.975632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:37.975910 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:37.976066 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:37.976457 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 22:20:37.976684 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:20:37.977003 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.964195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35818)
0927 22:20:37.964391 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.964415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:37.964552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:37.964911 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:37.964916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:37.965331 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:37.965511 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.965521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.966615 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.966622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.967468 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:37.967479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:37.968253 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:37.968259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:37.968472 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:37.969161 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:37.969179 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:37.971471 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:37.974580 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.974586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.974590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.974957 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.975359 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:37.975365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:37.975369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:37.975512 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:37.976031 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:37.976036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:37.976208 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 22:20:37.976351 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:20:37.976630 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:20:37.978127 14231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:37.979333 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.978935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54926)
0927 22:20:37.979111 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.979197 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:20:37.980547 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:37.980186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54936)
0927 22:20:37.980346 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:37.980451 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:20:37.980837 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:38.027313 14305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 3
I20260927 22:20:38.028515 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.028044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54940)
0927 22:20:38.028269 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.028361 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:20:38.029663 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.029253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54954)
0927 22:20:38.029473 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.029545 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260927 22:20:38.029922 14305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:38.032066 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 1
I20260927 22:20:38.033103 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.032694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54960)
0927 22:20:38.032882 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.032960 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:20:38.034087 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.033704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54976)
0927 22:20:38.033851 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.033926 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:20:38.034278 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:38.102485 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 2
I20260927 22:20:38.103690 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.103141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54978)
0927 22:20:38.103358 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.103504 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:20:38.104792 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.104338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54992)
0927 22:20:38.104538 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.104636 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:20:38.105042 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
W20260927 22:20:38.159544 14312 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:38.159847 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:38.159919 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:38.160000 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:38.167079 14312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:38.167161 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:38.167218 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:38.167268 14312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:38.167339 14312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:38.173223 14305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 4
I20260927 22:20:38.174655 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.174042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:54996)
0927 22:20:38.174312 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.174407 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:20:38.175820 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.175401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55004)
0927 22:20:38.175640 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.175716 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:20:38.176208 14305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:38.178164 14312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:37857,127.23.173.125:46213,127.23.173.124:39567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:39567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:38.179414 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:38.180810 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:38.192677 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:38.192628 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:38.193145 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:38.193799 14312 server_base.cc:1034] running on GCE node
I20260927 22:20:38.194777 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:38.196204 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:38.197430 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790547638197362 us; error 33 us; skew 500 ppm
I20260927 22:20:38.197836 14312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547638, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:38.201154 14312 init.cc:377] Logged in from keytab as kudu/127.23.173.124@KRBTEST.COM (short username kudu)
I20260927 22:20:38.202682 14312 webserver.cc:460] Webserver started at http://127.23.173.124:39295/ using document root <none> and password file <none>
I20260927 22:20:38.203411 14312 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:38.203562 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:38.203922 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:38.207302 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data/instance:
uuid: "9d8c346ea7d04c70b226ea98fa34a5a4"
format_stamp: "Formatted at 2026-09-27 22:20:38 on dist-test-slave-3wc3"
server_key: "190c3ae2bdaacc81686c07a6ab5e7b84"
server_key_iv: "11a396a4b64591e598813fcb20795765"
server_key_version: "encryptionkey@0"
I20260927 22:20:38.208165 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal/instance:
uuid: "9d8c346ea7d04c70b226ea98fa34a5a4"
format_stamp: "Formatted at 2026-09-27 22:20:38 on dist-test-slave-3wc3"
server_key: "190c3ae2bdaacc81686c07a6ab5e7b84"
server_key_iv: "11a396a4b64591e598813fcb20795765"
server_key_version: "encryptionkey@0"
I20260927 22:20:38.211259 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 3
I20260927 22:20:38.212373 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.211875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55016)
0927 22:20:38.212081 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.212163 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:20:38.213452 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.213057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55020)
0927 22:20:38.213238 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.213310 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 22:20:38.213686 14231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:39567: Network error: Client connection negotiation failed: client connection to 127.23.173.124:39567: connect: Connection refused (error 111)
I20260927 22:20:38.214329 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:20:38.217891 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:38.219364 14312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:38.219548 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "9d8c346ea7d04c70b226ea98fa34a5a4"
format_stamp: "Formatted at 2026-09-27 22:20:38 on dist-test-slave-3wc3"
server_key: "190c3ae2bdaacc81686c07a6ab5e7b84"
server_key_iv: "11a396a4b64591e598813fcb20795765"
server_key_version: "encryptionkey@0"
I20260927 22:20:38.219792 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:38.241178 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:38.244903 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:38.245170 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:38.264186 14312 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.124:39567
I20260927 22:20:38.264214 14378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.124:39567 every 8 connection(s)
I20260927 22:20:38.265599 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
I20260927 22:20:38.266387 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14312
I20260927 22:20:38.266601 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790547283249637-24245-0/minicluster-data/master-2/wal/instance
I20260927 22:20:38.267218 24245 external_mini_cluster.cc:1428] Setting key 332610c89780e6ab42462d8c817451ae
I20260927 22:20:38.270363 14379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:38.277136 14379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547638, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:20:38.290639 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.269138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.23.173.124:39567)
0927 22:20:38.269867 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.269879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.269917 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:20:38.271682 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.271704 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.272057 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.272446 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.273850 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.275344 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.276668 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.277348 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.277547 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.282076 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.282083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.282093 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.282126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.286397 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.287214 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.287223 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.287232 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.287449 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.287791 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.287796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.287799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.288339 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.288544 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.289058 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.289411 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":328,"threads_started":1}
I20260927 22:20:38.294008 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.279588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:37857 (local address 127.0.0.1:47274)
0927 22:20:38.280300 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.280342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:38.280495 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:38.280853 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:38.280858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:38.281511 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:38.281753 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.281762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.282827 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.282834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.283843 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.283854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.284559 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.284565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.284749 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.285317 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:38.285333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:38.287857 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:38.291176 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.291186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.291192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.291716 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.292204 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.292207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.292210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.292332 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.292968 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:38.292974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:38.293176 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 22:20:38.293390 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:20:38.293636 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":333,"threads_started":1}
I20260927 22:20:38.294571 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.280040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.23.173.126:37857)
0927 22:20:38.280494 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.280501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.280524 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:20:38.280591 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.280596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.280680 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.280796 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.281924 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.282660 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.283969 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.284375 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.284515 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.288038 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.288042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.288045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.288078 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.291065 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.291874 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.291878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.291880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.292077 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.292442 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.292446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.292449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.292830 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.292995 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.293828 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.294106 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:38.295970 14379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547638, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:38.310554 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.296797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:46213 (local address 127.0.0.1:35822)
0927 22:20:38.297041 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.297055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:38.297171 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:38.297481 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:38.297485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:38.297842 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:38.298048 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.298058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.298980 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.298987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.299838 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.299849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.300507 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.300513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.300709 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.301230 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:38.301245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:38.303871 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:38.307736 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.307742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.307747 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.308257 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.308689 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.308694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.308697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.308863 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.309508 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:38.309513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:38.309690 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 22:20:38.309850 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:20:38.310166 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 22:20:38.310633 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.296796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.23.173.125:46213)
0927 22:20:38.296977 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.296983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.297131 (+ 148us) server_negotiation.cc:407] Connection header received
0927 22:20:38.297298 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.297302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.297361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.297460 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.298186 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.298851 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.299969 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.300362 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.300493 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.304074 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.304081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.304102 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.304143 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.307598 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.308394 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.308402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.308405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.308556 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.308981 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.308986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.308989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.309379 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.309530 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.310040 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.310259 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:20:38.310637 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.294051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.23.173.126:37857)
0927 22:20:38.295031 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.295035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.295048 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:20:38.295089 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.295092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.295141 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.295255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.296840 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.297586 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.298866 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.299274 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.299430 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.303747 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.303752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.303755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.303786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.307428 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.308286 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.308289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.308292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.308488 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.308992 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.308996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.309000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.309379 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.309685 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.310032 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.310260 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":1345,"threads_started":1}
I20260927 22:20:38.311800 14379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547638, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:38.325098 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.312686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55040)
0927 22:20:38.313031 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.313049 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:38.313175 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:38.313452 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:38.313456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:38.313831 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:38.314003 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.314018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.314949 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.314957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.315791 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.315802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.316458 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.316463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.316666 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.317254 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:38.317280 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:38.319338 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:38.322392 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.322399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.322405 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.322922 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.323332 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.323337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.323342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.323501 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.324055 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:38.324059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:38.324213 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 22:20:38.324348 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:20:38.324654 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260927 22:20:38.325176 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.312874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.23.173.124:39567)
0927 22:20:38.313033 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.313040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.313117 (+ 77us) server_negotiation.cc:407] Connection header received
0927 22:20:38.313293 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.313296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.313343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.313436 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.314142 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.314830 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.315923 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.316310 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.316444 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.319506 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.319511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.319513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.319540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.322256 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.323094 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.323098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.323100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.323215 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.323627 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.323630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.323632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.323939 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.324141 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.324420 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.324812 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":52}
I20260927 22:20:38.327163 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.312749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.23.173.125:46213)
0927 22:20:38.312920 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.312925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.312941 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:20:38.313057 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.313062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.313125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.313220 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.314439 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.315120 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.316249 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.316655 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.316785 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.321336 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.321341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.321344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.321374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.324456 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.325141 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.325148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.325153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.325351 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.325710 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.325715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.325719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.326084 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.326316 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.326606 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.326847 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:20:38.331416 14379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: Bootstrap starting.
I20260927 22:20:38.332461 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:38.334872 14379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:38.336248 14379 log.cc:824] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:38.335537 14206 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "89c0b27b48824208adffc5fccd2c60d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47278
I20260927 22:20:38.339433 14379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: No bootstrap required, opened a new log
I20260927 22:20:38.350092 14379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.350498 14379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8c346ea7d04c70b226ea98fa34a5a4, State: Initialized, Role: FOLLOWER
I20260927 22:20:38.350992 14379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.351884 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.352257 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.353134 14379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:20:38.362454 14397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:20:38.362578 14397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:20:38.360528 14312 master_runner.cc:186] Error getting master registration for 127.23.173.126:37857: OK, instance_id { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" instance_seqno: 1790547637250578 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:20:38.371779 14312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:20:38.390920 14231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 4
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
I20260927 22:20:38.407287 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.391736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.23.173.124:39567)
0927 22:20:38.391971 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.391977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.392118 (+ 141us) server_negotiation.cc:407] Connection header received
0927 22:20:38.392307 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.392313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.392396 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.392529 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.393490 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.394698 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.396497 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.397191 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.397320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.401374 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.401381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.401386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.401428 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.404541 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.405206 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.405218 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.405222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.405338 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.405739 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.405745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.405749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.406151 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.406331 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.406702 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.406914 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:20:38.407298 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.391703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55046)
0927 22:20:38.392013 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.392036 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:38.392176 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:38.392568 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:38.392574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:38.393121 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:38.393330 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.393342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.394833 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.394839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.396341 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.396358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.397346 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.397352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.397563 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.398487 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:38.398507 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:38.401164 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:38.404693 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.404698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.404702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.405073 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.405458 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.405463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.405467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.405618 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.406267 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:38.406271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:38.406454 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 22:20:38.406598 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:20:38.406891 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260927 22:20:38.411989 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: Bootstrap starting.
I20260927 22:20:38.416115 14231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:38.417392 14231 log.cc:824] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:38.420724 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: No bootstrap required, opened a new log
I20260927 22:20:38.427616 14231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.428013 14231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c0b27b48824208adffc5fccd2c60d4, State: Initialized, Role: FOLLOWER
I20260927 22:20:38.428493 14231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.429486 14400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.429888 14400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.430708 14231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:38.430926 14206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:20:38.431143 14206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:38.434984 14206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.437371 14206 leader_election.cc:288] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [CANDIDATE]: Term 1 election: Requested vote from peers d2cf883504a34b81b76189b9620ac1ff (127.23.173.125:46213), 9d8c346ea7d04c70b226ea98fa34a5a4 (127.23.173.124:39567)
I20260927 22:20:38.438150 14354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89c0b27b48824208adffc5fccd2c60d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4"
I20260927 22:20:38.438776 14354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:38.443423 14354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c0b27b48824208adffc5fccd2c60d4 in term 1.
I20260927 22:20:38.444167 14183 leader_election.cc:302] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89c0b27b48824208adffc5fccd2c60d4, 9d8c346ea7d04c70b226ea98fa34a5a4; no voters:
I20260927 22:20:38.444705 14400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:38.453665 14400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [term 1 LEADER]: Becoming Leader. State: Replica: 89c0b27b48824208adffc5fccd2c60d4, State: Running, Role: LEADER
I20260927 22:20:38.455199 14400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.475479 14305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } attempt: 5
I20260927 22:20:38.479032 14408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c0b27b48824208adffc5fccd2c60d4. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.479629 14408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: This master's current role is: LEADER
Sep 27 22:20:38 dist-test-slave-3wc3 krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
W20260927 22:20:38.490216 14185 tablet.cc:2369] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:20:38.496872 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.476187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:39567 (local address 127.0.0.1:55056)
0927 22:20:38.476635 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:38.476661 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:38.476830 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:38.477893 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:38.477899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:38.478444 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:38.478660 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.478672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.480408 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.480413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.481947 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:38.481983 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.483363 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.483371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:38.483721 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.484489 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:38.484519 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:38.488076 (+ 3557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:38.493332 (+ 5256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.493339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.493344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.493993 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.494514 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:38.494522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:38.494528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:38.494775 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:38.495586 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:38.495594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:38.495802 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 22:20:38.495963 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:20:38.496335 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
W20260927 22:20:38.497349 14333 tablet.cc:2369] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:20:38.497515 14417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:38.499181 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.477122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.23.173.124:39567)
0927 22:20:38.477353 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.477360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.477386 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:20:38.477467 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.477476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.477572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.477690 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.478840 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.480268 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.482166 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.482970 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.483113 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.488347 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.488357 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.488364 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.488413 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.493141 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.494156 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.494163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.494168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.494357 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.494925 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.494933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.494936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.495342 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.495507 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.498401 (+ 2894us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.498704 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:20:38.506131 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: Bootstrap starting.
I20260927 22:20:38.514557 14305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:38.516098 14305 log.cc:824] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:38.517696 14417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:38.528193 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.466010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.23.173.126:37857)
0927 22:20:38.466220 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.466227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.487778 (+ 21551us) server_negotiation.cc:407] Connection header received
0927 22:20:38.489862 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.489869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.490832 (+ 963us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.490927 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.498126 (+ 7199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.499375 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.505198 (+ 5823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.506030 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.506148 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.514420 (+ 8272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.514426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.514432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.514475 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.518904 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.520087 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.520093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.520097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.520318 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.526095 (+ 5777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.526102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.526106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.526470 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.526755 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.527388 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.527745 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:20:38.530843 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: No bootstrap required, opened a new log
I20260927 22:20:38.533139 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.470456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.23.173.125:46213)
0927 22:20:38.470699 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.470708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.494035 (+ 23327us) server_negotiation.cc:407] Connection header received
0927 22:20:38.495201 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.495206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.495292 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.495388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.501709 (+ 6321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.503036 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.509754 (+ 6718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.510537 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.510682 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.520905 (+ 10223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.520912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.520917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.520980 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.525055 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.530313 (+ 5258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.530319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.530323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.530526 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.531025 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.531031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.531035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.531451 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.531733 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.532420 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.532710 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:20:38.542858 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:38.482356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.23.173.124:39567)
0927 22:20:38.510245 (+ 27889us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.510254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.510279 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:20:38.510341 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.510348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.510437 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.513312 (+ 2875us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.513500 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.517559 (+ 4059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.519957 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:38.520859 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:38.525470 (+ 4611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:38.525717 (+ 247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:38.525723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:38.525727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:38.525777 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:38.538384 (+ 12607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.539374 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.539380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.539383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.539582 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:38.540106 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:38.540111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:38.540115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:38.540497 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:38.540817 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:38.542124 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0927 22:20:38.542399 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27694,"thread_start_us":54166,"threads_started":1}
I20260927 22:20:38.547653 14305 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.548053 14305 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cf883504a34b81b76189b9620ac1ff, State: Initialized, Role: FOLLOWER
I20260927 22:20:38.548614 14305 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } }
I20260927 22:20:38.550076 14354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [term 1 FOLLOWER]: Refusing update from remote peer 89c0b27b48824208adffc5fccd2c60d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:20:38.552285 14408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:20:38.557106 14426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.557514 14426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.559374 14305 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:38.559787 14279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:38.559401 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89c0b27b48824208adffc5fccd2c60d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d2cf883504a34b81b76189b9620ac1ff"
I20260927 22:20:38.566310 14279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [term 1 FOLLOWER]: Refusing update from remote peer 89c0b27b48824208adffc5fccd2c60d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:20:38.542362 14164 master_runner.cc:186] Error getting master registration for 127.23.173.125:46213: OK, instance_id { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" instance_seqno: 1790547637747510 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:20:38.567692 14164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:20:38.570909 14280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c0b27b48824208adffc5fccd2c60d4 in term 1.
I20260927 22:20:38.574335 14408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:20:38.594762 14426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c0b27b48824208adffc5fccd2c60d4. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.595072 14426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.596946 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c0b27b48824208adffc5fccd2c60d4. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.597285 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.599750 14423 mvcc.cc:205] Tried to move back new op lower bound from 7334083127489814528 to 7334083127125512192. Current Snapshot: MvccSnapshot[applied={T|T < 7334083127489814528}]
I20260927 22:20:38.604542 14427 mvcc.cc:205] Tried to move back new op lower bound from 7334083127489814528 to 7334083127125512192. Current Snapshot: MvccSnapshot[applied={T|T < 7334083127489814528}]
I20260927 22:20:38.615540 14433 mvcc.cc:205] Tried to move back new op lower bound from 7334083127489814528 to 7334083127125512192. Current Snapshot: MvccSnapshot[applied={T|T < 7334083127489814528}]
I20260927 22:20:38.630404 14400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.630828 14400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:38.631312 14400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.631616 14400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:38.632853 14417 catalog_manager.cc:1347] Generated new cluster ID: 068cece8320246849051cb716051dc4f
I20260927 22:20:38.632959 14417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:38.633565 14426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.633850 14426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.646863 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89c0b27b48824208adffc5fccd2c60d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c0b27b48824208adffc5fccd2c60d4" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 37857 } } peers { permanent_uuid: "d2cf883504a34b81b76189b9620ac1ff" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 46213 } } peers { permanent_uuid: "9d8c346ea7d04c70b226ea98fa34a5a4" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 39567 } } }
I20260927 22:20:38.647135 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:38.655821 14417 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:38.658027 14417 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:38.665066 14445 catalog_manager.cc:1259] Loaded cluster ID: 068cece8320246849051cb716051dc4f
I20260927 22:20:38.665210 14445 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: loading cluster ID for follower catalog manager: success
I20260927 22:20:38.668533 14445 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: acquiring CA information for follower catalog manager: success
W20260927 22:20:38.680526 14445 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 22:20:38.682760 14417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c0b27b48824208adffc5fccd2c60d4: Generated new TSK 0
I20260927 22:20:38.683931 14417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:20:38.817490 14235 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.795245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.23.173.126:37857)
0927 22:20:38.795478 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.795489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.814742 (+ 19253us) server_negotiation.cc:407] Connection header received
0927 22:20:38.815131 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.815141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.815282 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.815386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.817104 (+ 1718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47302: BlockingRecv error: recv got EOF from 127.0.0.1:47302 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:20:38.820873 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.801728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.23.173.125:46213)
0927 22:20:38.801914 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.801920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.818595 (+ 16675us) server_negotiation.cc:407] Connection header received
0927 22:20:38.819387 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.819395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.819485 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.819598 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.820619 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35856: BlockingRecv error: recv got EOF from 127.0.0.1:35856 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:20:38.821002 14382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.806492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.23.173.124:39567)
0927 22:20:38.806701 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.806708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.819119 (+ 12411us) server_negotiation.cc:407] Connection header received
0927 22:20:38.819332 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.819341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.819436 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.819538 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.820738 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55074: BlockingRecv error: recv got EOF from 127.0.0.1:55074 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 22:20:38.868448 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.859527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.23.173.125:46213)
0927 22:20:38.859734 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.859746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.866537 (+ 6791us) server_negotiation.cc:407] Connection header received
0927 22:20:38.866827 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.866833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.866964 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.867058 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.868143 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35858: BlockingRecv error: recv got EOF from 127.0.0.1:35858 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:20:38.909521 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.899388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.23.173.125:46213)
0927 22:20:38.899593 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.899599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.907028 (+ 7429us) server_negotiation.cc:407] Connection header received
0927 22:20:38.907526 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.907532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.907639 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.907737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.909280 (+ 1543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35860: BlockingRecv error: recv got EOF from 127.0.0.1:35860 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:20:38.957621 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.947665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.23.173.125:46213)
0927 22:20:38.947874 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.947880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:38.955552 (+ 7672us) server_negotiation.cc:407] Connection header received
0927 22:20:38.955926 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:38.955933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:38.956055 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:38.956150 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:38.957340 (+ 1190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35876: BlockingRecv error: recv got EOF from 127.0.0.1:35876 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 22:20:39.004279 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:38.994441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.23.173.125:46213)
0927 22:20:38.994712 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:38.994719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.002203 (+ 7484us) server_negotiation.cc:407] Connection header received
0927 22:20:39.002728 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.002734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.002842 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.002927 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.004024 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35884: BlockingRecv error: recv got EOF from 127.0.0.1:35884 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260927 22:20:39.057781 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.047677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.23.173.125:46213)
0927 22:20:39.047891 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.047898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.055550 (+ 7652us) server_negotiation.cc:407] Connection header received
0927 22:20:39.056173 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.056182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.056292 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.056381 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.057512 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35896: BlockingRecv error: recv got EOF from 127.0.0.1:35896 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 22:20:39.123047 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.113038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.23.173.125:46213)
0927 22:20:39.113247 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.113254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.121024 (+ 7770us) server_negotiation.cc:407] Connection header received
0927 22:20:39.121534 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.121541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.121636 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.121712 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.122780 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35906: BlockingRecv error: recv got EOF from 127.0.0.1:35906 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:20:39.161799 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.159304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.23.173.125:46213)
0927 22:20:39.159569 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.159583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.160257 (+ 674us) server_negotiation.cc:407] Connection header received
0927 22:20:39.160487 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.160498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.160572 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.160688 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.161504 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35912: BlockingRecv error: recv got EOF from 127.0.0.1:35912 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:20:39.231695 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.229506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.23.173.125:46213)
0927 22:20:39.229736 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.229743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.230158 (+ 415us) server_negotiation.cc:407] Connection header received
0927 22:20:39.230343 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.230350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.230457 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.230588 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.231423 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35924: BlockingRecv error: recv got EOF from 127.0.0.1:35924 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:20:39.365299 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.363091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.23.173.125:46213)
0927 22:20:39.363338 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.363348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.363892 (+ 544us) server_negotiation.cc:407] Connection header received
0927 22:20:39.364129 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.364134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.364205 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.364311 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.365050 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35928: BlockingRecv error: recv got EOF from 127.0.0.1:35928 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:20:39.365324 14397 catalog_manager.cc:1259] Loaded cluster ID: 068cece8320246849051cb716051dc4f
I20260927 22:20:39.365424 14397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: loading cluster ID for follower catalog manager: success
I20260927 22:20:39.369750 14397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: acquiring CA information for follower catalog manager: success
I20260927 22:20:39.372545 14397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d8c346ea7d04c70b226ea98fa34a5a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:20:39.627142 14309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:39.625273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.23.173.125:46213)
0927 22:20:39.625449 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:39.625454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:39.625728 (+ 274us) server_negotiation.cc:407] Connection header received
0927 22:20:39.625912 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:39.625916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:39.626010 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:39.626102 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:39.626886 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57180: BlockingRecv error: recv got EOF from 127.0.0.1:57180 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:20:39.683400 14445 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2cf883504a34b81b76189b9620ac1ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:20:40.149727 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:40.143620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.23.173.125:46213)
0927 22:20:40.144162 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:40.144169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:40.144284 (+ 115us) server_negotiation.cc:407] Connection header received
0927 22:20:40.144538 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:40.144542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:40.144694 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:40.144831 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:40.145277 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.146076 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.147433 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.147863 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.148000 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:40.148531 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:40.149087 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:40.149249 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:40.149355 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 22:20:40.149441 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260927 22:20:40.162036 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:40.155835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.23.173.124:39567)
0927 22:20:40.156656 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:40.156662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:40.156678 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:20:40.156770 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:40.156773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:40.156994 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:40.157168 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:40.157542 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.158326 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.159771 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.160198 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.160353 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:40.160844 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:40.161379 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:40.161544 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:40.161601 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 22:20:40.161684 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":575,"threads_started":1}
I20260927 22:20:40.186648 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:40.168672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.23.173.125:46213)
0927 22:20:40.168904 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:40.168913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:40.180628 (+ 11715us) server_negotiation.cc:407] Connection header received
0927 22:20:40.180928 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:40.180934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:40.181036 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:40.181133 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:40.181684 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.182633 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.184597 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.185200 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.185344 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:40.185816 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:40.185948 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:40.186096 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:40.186240 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:20:40.186320 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:20:40.187110 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:40.168604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.23.173.124:39567)
0927 22:20:40.174540 (+ 5936us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:40.174548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:40.181037 (+ 6489us) server_negotiation.cc:407] Connection header received
0927 22:20:40.181254 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:40.181262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:40.181340 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:40.181390 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:40.182070 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.183047 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.185101 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.185676 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.185792 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:40.186338 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:40.186450 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:40.186510 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:40.186758 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 22:20:40.186841 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5777}
I20260927 22:20:40.187861 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:40.168311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.23.173.126:37857)
0927 22:20:40.179470 (+ 11159us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:40.179480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:40.179871 (+ 391us) server_negotiation.cc:407] Connection header received
0927 22:20:40.180148 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:40.180156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:40.180394 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:40.181249 (+ 855us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:40.181432 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.182412 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.184718 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:40.185358 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:40.185480 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:40.186051 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:40.186506 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:40.186622 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:40.187432 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 22:20:40.187515 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10984,"thread_start_us":10859,"threads_started":1}
I20260927 22:20:40.192938 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14164
I20260927 22:20:40.194156 14226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:40.308343 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14164
I20260927 22:20:40.326231 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14238
I20260927 22:20:40.327325 14300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:40.435652 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14238
I20260927 22:20:40.456246 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14312
I20260927 22:20:40.457360 14374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:40.564729 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14312
2026-09-27T22:20:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5130 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:20:40 dist-test-slave-3wc3 krb5kdc[14531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:20:40 dist-test-slave-3wc3 krb5kdc[14531](info): set up 2 sockets
Sep 27 22:20:40 dist-test-slave-3wc3 krb5kdc[14531](info): commencing operation
krb5kdc: starting...
W20260927 22:20:41.686564 24245 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/test-user.keytab.
Sep 27 22:20:41 dist-test-slave-3wc3 krb5kdc[14531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.173.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
Entry for principal kudu/127.23.173.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab.
I20260927 22:20:41.945420 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:44509
--webserver_interface=127.23.173.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:20:42.276130 14544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:42.276444 14544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:42.276517 14544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:42.276600 14544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:42.283792 14544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:42.283886 14544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:42.283943 14544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:42.283995 14544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:42.284049 14544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:42.293849 14544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.126.keytab
--principal=kudu/127.23.173.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.126:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.173.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:42.295110 14544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:42.296326 14544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:42.308264 14550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:42.308350 14552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:42.308382 14549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:42.309742 14544 server_base.cc:1034] running on GCE node
I20260927 22:20:42.310598 14544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:42.311949 14544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:42.313201 14544 hybrid_clock.cc:648] HybridClock initialized: now 1790547642313139 us; error 37 us; skew 500 ppm
I20260927 22:20:42.313529 14544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:42 dist-test-slave-3wc3 krb5kdc[14531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:42.316751 14544 init.cc:377] Logged in from keytab as kudu/127.23.173.126@KRBTEST.COM (short username kudu)
I20260927 22:20:42.318506 14544 webserver.cc:460] Webserver started at http://127.23.173.126:38059/ using document root <none> and password file <none>
I20260927 22:20:42.319295 14544 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:42.319389 14544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:42.319682 14544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:42.322738 14544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data/instance:
uuid: "d95e93b0be4641d3b34318f4f617c41b"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "b82170a321f98904cfcaa43d687b6d3e"
server_key_iv: "4d4be0e4fee91793f2c3eab39de77561"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.323499 14544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance:
uuid: "d95e93b0be4641d3b34318f4f617c41b"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "b82170a321f98904cfcaa43d687b6d3e"
server_key_iv: "4d4be0e4fee91793f2c3eab39de77561"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.328593 14544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:20:42.331966 14559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:42.333240 14544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:42.333436 14544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal
uuid: "d95e93b0be4641d3b34318f4f617c41b"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "b82170a321f98904cfcaa43d687b6d3e"
server_key_iv: "4d4be0e4fee91793f2c3eab39de77561"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.333652 14544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:42.351707 14544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:42.355816 14544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:42.356106 14544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:42.374976 14544 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.126:44509
I20260927 22:20:42.374997 14610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.126:44509 every 8 connection(s)
I20260927 22:20:42.376451 14544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/data/info.pb
I20260927 22:20:42.380153 14611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:42.381167 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14544
I20260927 22:20:42.381371 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-0/wal/instance
I20260927 22:20:42.381832 24245 external_mini_cluster.cc:1428] Setting key 920b5a890bd3a32ee5e08e1742514714
I20260927 22:20:42.386797 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:42 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.173.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.125@KRBTEST.COM" created.
I20260927 22:20:42.408905 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.389934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:44509 (local address 127.0.0.1:60410)
0927 22:20:42.391096 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.391131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:42.392764 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:42.393620 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:42.393627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:42.394376 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:42.394704 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.394725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.396275 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.396281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.397338 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.397350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.398315 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.398322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.398501 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.399171 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:42.399192 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:42.401490 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:42.404855 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.404868 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.404881 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.405318 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.405759 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.405763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.405766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.405933 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.406785 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:42.406795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:42.406984 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 22:20:42.407197 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 22:20:42.407930 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":168,"thread_start_us":406,"threads_started":1}
I20260927 22:20:42.408941 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.389933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.23.173.126:44509)
0927 22:20:42.391065 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:42.391078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:42.391240 (+ 162us) server_negotiation.cc:407] Connection header received
0927 22:20:42.393136 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:42.393141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:42.393397 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:42.393598 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:42.394934 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.396113 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.397479 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.398086 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.398296 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.401725 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:42.401733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:42.401746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:42.401774 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:42.404735 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.405446 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.405454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.405458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.405610 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.406086 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.406090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.406092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.406560 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:42.406825 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:42.407369 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 22:20:42.407726 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"spinlock_wait_cycles":32000,"thread_start_us":621,"threads_started":1}
I20260927 22:20:42.412056 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:42.413194 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.412770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33222)
0927 22:20:42.412937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.413067 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:20:42.413453 14563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
I20260927 22:20:42.414961 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.414628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33238)
0927 22:20:42.414798 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.414858 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:20:42.415144 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:37475: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
Entry for principal kudu/127.23.173.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab.
I20260927 22:20:42.437455 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:37475
--webserver_interface=127.23.173.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:20:42.464491 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } attempt: 1
I20260927 22:20:42.465708 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.465224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33242)
0927 22:20:42.465440 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.465567 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:20:42.466847 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.466489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33248)
0927 22:20:42.466642 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.466716 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:20:42.467041 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:37475: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
I20260927 22:20:42.535322 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } attempt: 2
I20260927 22:20:42.536540 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.536040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33252)
0927 22:20:42.536306 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.536398 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:20:42.537611 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.537264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33256)
0927 22:20:42.537415 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.537493 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:20:42.537865 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:37475: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
I20260927 22:20:42.629213 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } attempt: 3
I20260927 22:20:42.630596 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.630140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33270)
0927 22:20:42.630359 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.630455 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:20:42.631764 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.631438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33278)
0927 22:20:42.631571 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.631639 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:20:42.632067 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:37475: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
W20260927 22:20:42.767066 14618 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:42.767438 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:42.767562 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:42.767706 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:42.775266 14618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:42.775377 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260927 22:20:42.775321 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } attempt: 4
W20260927 22:20:42.775493 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:42.775597 14618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:42.775713 14618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:42.776659 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.776132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33284)
0927 22:20:42.776437 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.776526 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 22:20:42.777638 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.777279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33298)
0927 22:20:42.777439 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.777518 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:20:42.777806 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.125:37475: Network error: Client connection negotiation failed: client connection to 127.23.173.125:37475: connect: Connection refused (error 111)
I20260927 22:20:42.786629 14618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.125.keytab
--principal=kudu/127.23.173.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.125:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.173.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:42.788002 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:42.789354 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:42.801613 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:42.801668 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:42.801697 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:42.811537 14618 server_base.cc:1034] running on GCE node
I20260927 22:20:42.812361 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:42.813851 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:42.815124 14618 hybrid_clock.cc:648] HybridClock initialized: now 1790547642815052 us; error 47 us; skew 500 ppm
I20260927 22:20:42.815472 14618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:42 dist-test-slave-3wc3 krb5kdc[14531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:42.818763 14618 init.cc:377] Logged in from keytab as kudu/127.23.173.125@KRBTEST.COM (short username kudu)
I20260927 22:20:42.820303 14618 webserver.cc:460] Webserver started at http://127.23.173.125:42913/ using document root <none> and password file <none>
I20260927 22:20:42.821079 14618 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:42.821225 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:42.821544 14618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:42.824536 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data/instance:
uuid: "8b8d8ca8e1a54ab693686cc32240b725"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "801b5a7a4054149ad7b353491a7383fa"
server_key_iv: "2b274be1d7c8a7859f52709ee4e7b554"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.825343 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal/instance:
uuid: "8b8d8ca8e1a54ab693686cc32240b725"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "801b5a7a4054149ad7b353491a7383fa"
server_key_iv: "2b274be1d7c8a7859f52709ee4e7b554"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.830468 14618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:20:42.833904 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:42.835263 14618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:20:42.835445 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal
uuid: "8b8d8ca8e1a54ab693686cc32240b725"
format_stamp: "Formatted at 2026-09-27 22:20:42 on dist-test-slave-3wc3"
server_key: "801b5a7a4054149ad7b353491a7383fa"
server_key_iv: "2b274be1d7c8a7859f52709ee4e7b554"
server_key_version: "encryptionkey@0"
I20260927 22:20:42.835683 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:42.858906 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:42.863027 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:42.863317 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:42.881675 14618 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.125:37475
I20260927 22:20:42.881701 14684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.125:37475 every 8 connection(s)
I20260927 22:20:42.883087 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/data/info.pb
I20260927 22:20:42.886821 14685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:42.892562 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14618
I20260927 22:20:42.892752 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-1/wal/instance
I20260927 22:20:42.893268 24245 external_mini_cluster.cc:1428] Setting key aa3170506a7e3eb0fd9979633059a9d0
I20260927 22:20:42.893914 14685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:42 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:20:42.918406 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.897359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.23.173.126:44509)
0927 22:20:42.897598 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:42.897607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:42.898895 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 22:20:42.901842 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:42.901846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:42.901916 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:42.902054 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:42.904209 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.905377 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.906985 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.907733 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.907874 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.911327 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:42.911336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:42.911343 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:42.911389 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:42.915068 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.915824 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.915833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.915840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.915987 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.916388 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.916396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.916402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.916824 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:42.916978 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:42.917714 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 22:20:42.917994 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:20:42.919374 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.897815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:44509 (local address 127.0.0.1:60422)
0927 22:20:42.898688 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.898738 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:42.901521 (+ 2783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:42.902325 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:42.902338 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:42.903146 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:42.903997 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.904026 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.905523 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.905527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.906827 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.906839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.907934 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.907938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.908132 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.908840 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:42.908863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:42.911094 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:42.915238 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.915245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.915257 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.915687 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.916092 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.916095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.916097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.916265 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.916942 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:42.916948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:42.917271 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 22:20:42.917543 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 22:20:42.918149 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":363,"threads_started":1}
I20260927 22:20:42.921258 14685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.173.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.173.124@KRBTEST.COM" created.
Sep 27 22:20:42 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:42.938187 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.922290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.23.173.125:37475)
0927 22:20:42.922871 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:42.922877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:42.922924 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:20:42.922982 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:42.922993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:42.923188 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:42.923349 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:42.924184 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.925741 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.926934 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.927872 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.928013 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.931327 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:42.931346 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:42.931367 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:42.931409 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:42.935100 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.935964 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.935971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.935973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.936088 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:42.936507 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:42.936511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:42.936513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:42.936872 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:42.937040 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:42.937526 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 22:20:42.937759 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":433,"thread_start_us":278,"threads_started":1}
I20260927 22:20:42.938236 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.922213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33300)
0927 22:20:42.922385 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.922411 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:42.922553 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:42.923345 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:42.923349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:42.923842 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:42.924003 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.924011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.925902 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.925906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.926786 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:42.926798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:42.928014 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:42.928017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:42.928215 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:42.928807 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:42.928824 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:42.931083 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:42.935245 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.935252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.935257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.935831 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.936193 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:42.936201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:42.936207 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:42.936395 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:42.936996 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:42.937004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:42.937189 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:20:42.937335 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 22:20:42.937667 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":33}
I20260927 22:20:42.940938 14685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:42.942193 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.941714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57338)
0927 22:20:42.941890 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.942046 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:20:42.942476 14636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:42.944083 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.943731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57352)
0927 22:20:42.943912 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.943985 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:20:42.944322 14685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
Entry for principal kudu/127.23.173.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab.
I20260927 22:20:42.960739 24245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
/tmp/dist-test-taskO6uUAX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:45435
--webserver_interface=127.23.173.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.173.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:20:42.993615 14685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 1
I20260927 22:20:42.994867 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.994385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57354)
0927 22:20:42.994582 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.994692 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:20:42.996045 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:42.995621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57360)
0927 22:20:42.995788 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:42.995891 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:20:42.996300 14685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.064565 14685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 2
I20260927 22:20:43.065789 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.065329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57364)
0927 22:20:43.065528 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.065622 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:20:43.066821 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.066498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57376)
0927 22:20:43.066622 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.066690 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:20:43.067091 14685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.077040 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } attempt: 5
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:43.092815 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.077810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.23.173.125:37475)
0927 22:20:43.078021 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.078027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.078051 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:20:43.078237 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.078243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.078321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.078429 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.079156 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.080225 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.081350 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.082233 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.082390 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.085463 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.085471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.085476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.085520 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.090248 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.091006 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.091010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.091012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.091134 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.091562 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.091565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.091567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.091844 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.091954 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.092413 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.092592 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:43.093097 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.077702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33306)
0927 22:20:43.077913 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.077930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.078085 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.078463 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.078467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.078828 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.079009 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.079018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.080350 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.080354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.081192 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.081203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.082384 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.082387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.082552 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.083207 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.083223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.085240 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.090376 (+ 5136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.090380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.090383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.090828 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.091263 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.091267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.091271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.091454 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.091968 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.091973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.092148 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.092322 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.092626 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:20:43.094216 14611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:20:43.095249 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.094891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57380)
0927 22:20:43.095048 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.095123 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:20:43.096158 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.095853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57388)
0927 22:20:43.095989 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.096052 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:20:43.096385 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.147598 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 1
I20260927 22:20:43.148943 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.148435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57402)
0927 22:20:43.148671 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.148770 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 22:20:43.150103 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.149691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57418)
0927 22:20:43.149840 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.149918 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:20:43.150301 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.158315 14685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 3
I20260927 22:20:43.159468 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.159033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57420)
0927 22:20:43.159222 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.159313 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:20:43.160527 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.160117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57436)
0927 22:20:43.160339 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.160425 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260927 22:20:43.160777 14685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.218519 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 2
I20260927 22:20:43.219899 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.219261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57452)
0927 22:20:43.219556 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.219679 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 22:20:43.221002 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.220571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57460)
0927 22:20:43.220760 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.220849 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:20:43.221231 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
W20260927 22:20:43.292464 14692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:20:43.292819 14692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:20:43.292913 14692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:20:43.293043 14692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:20:43.300089 14692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:20:43.300170 14692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:20:43.300268 14692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:20:43.300355 14692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:20:43.300465 14692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:20:43.303963 14685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 4
I20260927 22:20:43.305190 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.304600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57468)
0927 22:20:43.304874 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.304993 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:20:43.306262 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.305850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57480)
0927 22:20:43.306027 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.306117 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:20:43.306483 14685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.310719 14692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.173.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.173.126:44509,127.23.173.125:37475,127.23.173.124:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/krb5kdc/kudu_127.23.173.124.keytab
--principal=kudu/127.23.173.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.173.124:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.173.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:20:43.311987 14692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:20:43.313294 14692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:20:43.325063 14698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:43.325155 14697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:20:43.325155 14700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:20:43.326321 14692 server_base.cc:1034] running on GCE node
I20260927 22:20:43.327136 14692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:20:43.327435 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 3
I20260927 22:20:43.328480 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.328038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57490)
0927 22:20:43.328252 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.328332 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:20:43.328812 14692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:20:43.329592 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.329232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57498)
0927 22:20:43.329412 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.329487 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 22:20:43.329834 14611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.173.124:45435: Network error: Client connection negotiation failed: client connection to 127.23.173.124:45435: connect: Connection refused (error 111)
I20260927 22:20:43.330080 14692 hybrid_clock.cc:648] HybridClock initialized: now 1790547643329976 us; error 64 us; skew 500 ppm
I20260927 22:20:43.330441 14692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547643, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:20:43.333595 14692 init.cc:377] Logged in from keytab as kudu/127.23.173.124@KRBTEST.COM (short username kudu)
I20260927 22:20:43.335050 14692 webserver.cc:460] Webserver started at http://127.23.173.124:34127/ using document root <none> and password file <none>
I20260927 22:20:43.335760 14692 fs_manager.cc:359] Metadata directory not provided
I20260927 22:20:43.335903 14692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:20:43.336225 14692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:20:43.339265 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data/instance:
uuid: "09c3167713284fcb8f63b9121c7ee619"
format_stamp: "Formatted at 2026-09-27 22:20:43 on dist-test-slave-3wc3"
server_key: "37cb5758bee767ff157180c717e1b5ad"
server_key_iv: "e203ddd127fe0cac24d84c0bf51dfa2f"
server_key_version: "encryptionkey@0"
I20260927 22:20:43.340094 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal/instance:
uuid: "09c3167713284fcb8f63b9121c7ee619"
format_stamp: "Formatted at 2026-09-27 22:20:43 on dist-test-slave-3wc3"
server_key: "37cb5758bee767ff157180c717e1b5ad"
server_key_iv: "e203ddd127fe0cac24d84c0bf51dfa2f"
server_key_version: "encryptionkey@0"
I20260927 22:20:43.345253 14692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:20:43.348603 14707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:20:43.349886 14692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:20:43.350132 14692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data,/tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal
uuid: "09c3167713284fcb8f63b9121c7ee619"
format_stamp: "Formatted at 2026-09-27 22:20:43 on dist-test-slave-3wc3"
server_key: "37cb5758bee767ff157180c717e1b5ad"
server_key_iv: "e203ddd127fe0cac24d84c0bf51dfa2f"
server_key_version: "encryptionkey@0"
I20260927 22:20:43.350350 14692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:20:43.361805 14692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:20:43.365610 14692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:20:43.365877 14692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:20:43.384465 14692 rpc_server.cc:307] RPC server started. Bound to: 127.23.173.124:45435
I20260927 22:20:43.384509 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.173.124:45435 every 8 connection(s)
I20260927 22:20:43.385736 14692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/data/info.pb
I20260927 22:20:43.386070 24245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu as pid 14692
I20260927 22:20:43.386252 24245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskO6uUAX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790547283249637-24245-0/minicluster-data/master-2/wal/instance
I20260927 22:20:43.386868 24245 external_mini_cluster.cc:1428] Setting key 1de17d7294cd4dd53f5baaed3dcb9f87
I20260927 22:20:43.390049 14759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:20:43.398391 14759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547643, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
I20260927 22:20:43.410000 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.388579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.23.173.124:45435)
0927 22:20:43.389357 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.389368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.389408 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:20:43.391059 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.391080 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.391523 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.391831 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.393543 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.394967 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.396850 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.397516 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.397640 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.402597 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.402619 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.402638 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.402692 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.406045 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.406731 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.406742 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.406753 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.406917 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.407304 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.407311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.407318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.407783 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.408130 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.408554 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.408938 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260927 22:20:43.417377 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.401472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.23.173.126:44509)
0927 22:20:43.401727 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.401734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.403079 (+ 1345us) server_negotiation.cc:407] Connection header received
0927 22:20:43.403302 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.403308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.403382 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.403486 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.404655 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.405344 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.406793 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.407223 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.407354 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.410943 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.410948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.410951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.410983 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.414631 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.415475 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.415479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.415481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.415614 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.415992 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.415997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.416000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.416313 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.416485 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.416914 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.417119 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:20:43.417421 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.401421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.126:44509 (local address 127.0.0.1:60430)
0927 22:20:43.402943 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.402987 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.403160 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.403521 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.403542 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.404237 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.404512 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.404521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.405500 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.405507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.406650 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.406662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.407413 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.407421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.407601 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.408205 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.408235 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.410751 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.414764 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.414778 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.414785 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.415356 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.415755 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.415759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.415761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.415882 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.416445 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.416451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.416599 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.416748 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.417011 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":1049,"threads_started":1}
I20260927 22:20:43.418845 14759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.126@KRBTEST.COM
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547643, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
I20260927 22:20:43.429728 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.413257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.23.173.126:44509)
0927 22:20:43.413991 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.413999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.414032 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:20:43.414086 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.414091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.414172 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.414301 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.415735 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.416895 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.418513 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.418906 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.419051 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.423632 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.423636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.423640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.423669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.426874 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.427577 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.427583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.427587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.427726 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.428190 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.428195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.428198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.428561 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.428857 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.429045 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.429296 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":245,"threads_started":1}
I20260927 22:20:43.433525 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.419628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.23.173.125:37475)
0927 22:20:43.419834 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.419841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.420123 (+ 282us) server_negotiation.cc:407] Connection header received
0927 22:20:43.420263 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.420267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.420327 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.420404 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.421290 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.421936 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.423275 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.423830 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.423925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.427255 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.427261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.427264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.427305 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.430622 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.431472 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.431477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.431481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.431704 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.432191 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.432197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.432200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.432569 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.432707 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.433114 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.433272 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:20:43.433728 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.419868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.125:37475 (local address 127.0.0.1:33318)
0927 22:20:43.420023 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.420042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.420151 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.420463 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.420469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.420936 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.421155 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.421168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.422132 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.422139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.423121 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.423134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.423983 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.423989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.424195 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.424673 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.424696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.427077 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.430771 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.430779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.430784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.431301 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.431834 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.431839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.431843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.432035 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.432685 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.432690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.432875 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.433042 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.433334 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 22:20:43.434960 14759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.173.125@KRBTEST.COM
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547643, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.124@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
I20260927 22:20:43.448225 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.431742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.23.173.125:37475)
0927 22:20:43.432385 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.432390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.432412 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:20:43.432453 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.432458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.432522 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.432640 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.434250 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.435213 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.436594 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.437100 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.437231 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.442273 (+ 5042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.442279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.442282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.442312 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.445240 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.446081 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.446086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.446089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.446252 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.446611 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.446616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.446619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.446976 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.447258 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.447530 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.447755 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":284,"threads_started":1}
I20260927 22:20:43.448752 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.435712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.23.173.124:45435)
0927 22:20:43.435879 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.435887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.436043 (+ 156us) server_negotiation.cc:407] Connection header received
0927 22:20:43.436213 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.436217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.436265 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.436356 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.437006 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.437761 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.439539 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.439999 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.440090 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.443206 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.443211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.443214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.443240 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.445927 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.446720 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.446724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.446726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.446837 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.447226 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.447233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.447237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.447582 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.447736 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.448171 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.448516 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":62}
I20260927 22:20:43.449160 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.435737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57516)
0927 22:20:43.435961 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.435983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.436090 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.436368 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.436372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.436745 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.436895 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.436905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.437912 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.437918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.439398 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.439416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.440158 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.440164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.440386 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.440983 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.440999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.443033 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.446093 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.446099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.446102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.446603 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.446947 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.446953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.446956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.447113 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.447708 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.447713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.447857 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.448058 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.448929 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 22:20:43.453557 24245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:20:43.454727 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: Bootstrap starting.
I20260927 22:20:43.457749 14759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:43.455960 14586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d95e93b0be4641d3b34318f4f617c41b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60442
I20260927 22:20:43.458877 14759 log.cc:824] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:43.461891 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: No bootstrap required, opened a new log
I20260927 22:20:43.468719 14759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.469055 14759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c3167713284fcb8f63b9121c7ee619, State: Initialized, Role: FOLLOWER
I20260927 22:20:43.469493 14759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.470450 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.470880 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.471347 14759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:20:43.481451 14778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:20:43.481623 14778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:20:43.479126 14692 master_runner.cc:186] Error getting master registration for 127.23.173.126:44509: OK, instance_id { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" instance_seqno: 1790547642369563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:20:43.487987 14692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:20:43.507019 14611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 4
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.126@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
I20260927 22:20:43.523413 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.507792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.23.173.124:45435)
0927 22:20:43.507995 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.507999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.508022 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:20:43.508240 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.508244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.508308 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.508393 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.509386 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.510175 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.511970 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.512401 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.512539 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.516641 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.516647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.516654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.516700 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.520445 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.521239 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.521244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.521248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.521392 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.521812 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.521816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.521820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.522264 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.522429 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.522842 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.523037 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:20:43.523428 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.507717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57532)
0927 22:20:43.507941 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.507967 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.508103 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.508442 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.508447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.508980 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.509235 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.509252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.510315 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.510321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.511825 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.511842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.512586 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.512593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.512802 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.513491 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.513519 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.516422 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.520570 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.520576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.520581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.521102 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.521510 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.521515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.521519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.521694 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.522411 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.522416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.522578 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.522732 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.523020 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:20:43.528134 14611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: Bootstrap starting.
I20260927 22:20:43.531328 14611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:43.532393 14611 log.cc:824] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:43.535355 14611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: No bootstrap required, opened a new log
I20260927 22:20:43.540764 14611 raft_consensus.cc:348] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.541091 14611 raft_consensus.cc:729] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d95e93b0be4641d3b34318f4f617c41b, State: Initialized, Role: FOLLOWER
I20260927 22:20:43.541533 14611 consensus_queue.cc:260] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.542487 14781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.542917 14781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.543660 14611 sys_catalog.cc:564] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:43.543851 14586 raft_consensus.cc:482] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:20:43.544067 14586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:43.547401 14586 raft_consensus.cc:504] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.550011 14586 leader_election.cc:288] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [CANDIDATE]: Term 1 election: Requested vote from peers 8b8d8ca8e1a54ab693686cc32240b725 (127.23.173.125:37475), 09c3167713284fcb8f63b9121c7ee619 (127.23.173.124:45435)
I20260927 22:20:43.550437 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d95e93b0be4641d3b34318f4f617c41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09c3167713284fcb8f63b9121c7ee619"
I20260927 22:20:43.551067 14734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:43.558099 14734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d95e93b0be4641d3b34318f4f617c41b in term 1.
I20260927 22:20:43.559415 14562 leader_election.cc:302] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c3167713284fcb8f63b9121c7ee619, d95e93b0be4641d3b34318f4f617c41b; no voters:
I20260927 22:20:43.560084 14781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:20:43.561398 14781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [term 1 LEADER]: Becoming Leader. State: Replica: d95e93b0be4641d3b34318f4f617c41b, State: Running, Role: LEADER
I20260927 22:20:43.562209 14781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.582845 14786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d95e93b0be4641d3b34318f4f617c41b. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.583745 14786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:43.598305 14797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:20:43.605710 14685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } attempt: 5
W20260927 22:20:43.609099 14565 tablet.cc:2369] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:20:43.617787 14713 tablet.cc:2369] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:20:43.647714 14797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:20:43.648100 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.586276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.23.173.125:37475)
0927 22:20:43.586539 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.586549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.604421 (+ 17872us) server_negotiation.cc:407] Connection header received
0927 22:20:43.605168 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.605176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.605289 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.605369 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.615246 (+ 9877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.616444 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.625335 (+ 8891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.626062 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.626166 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.638590 (+ 12424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.638597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.638602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.638648 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.642424 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.644325 (+ 1901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.644331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.644356 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.644615 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.645621 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.645627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.645631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.646270 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.646604 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.647201 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.647588 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:20:43.654455 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.579652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.23.173.126:44509)
0927 22:20:43.579836 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.579845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.603926 (+ 24081us) server_negotiation.cc:407] Connection header received
0927 22:20:43.606850 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.606856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.606944 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.607015 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.613912 (+ 6897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.615045 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.623373 (+ 8328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.624122 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.624238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.632032 (+ 7794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.632040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.632046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.632093 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.635942 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.644703 (+ 8761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.644710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.644714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.644910 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.645636 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.645642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.645645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.646025 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.646272 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.653768 (+ 7496us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.654056 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:20:43.659754 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.599415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.23.173.124:45435)
0927 22:20:43.599607 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.599614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.611546 (+ 11932us) server_negotiation.cc:407] Connection header received
0927 22:20:43.611798 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.611804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.611882 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.611948 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.619996 (+ 8048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.621162 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.631698 (+ 10536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.632862 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.633048 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.647600 (+ 14552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.647608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.647613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.647655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.653296 (+ 5641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.657330 (+ 4034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.657340 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.657344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.657510 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.657987 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.657993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.657996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.658336 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.658594 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.659105 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.659345 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":63}
Sep 27 22:20:43 dist-test-slave-3wc3 krb5kdc[14531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790547642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.173.125@KRBTEST.COM for kudu/127.23.173.124@KRBTEST.COM
I20260927 22:20:43.676486 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.606522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.173.124:45435 (local address 127.0.0.1:57548)
0927 22:20:43.606965 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 22:20:43.606988 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:20:43.607133 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:20:43.625702 (+ 18569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:20:43.625711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:20:43.626505 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:20:43.626781 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.626797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.652657 (+ 25860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.652668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.654659 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:20:43.654680 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.656160 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.656168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:20:43.656544 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.657310 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:20:43.657340 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:20:43.660767 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:20:43.672829 (+ 12062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.672836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.672841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.673355 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.674494 (+ 1139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:20:43.674499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:20:43.674503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:20:43.674720 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:20:43.675427 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:20:43.675432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:20:43.675611 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 22:20:43.675751 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:20:43.676049 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260927 22:20:43.678437 14734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [term 1 FOLLOWER]: Refusing update from remote peer d95e93b0be4641d3b34318f4f617c41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:20:43.680230 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.607391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.23.173.124:45435)
0927 22:20:43.624662 (+ 17271us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.624673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.624737 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:20:43.624871 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.624880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.625097 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.627181 (+ 2084us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.628837 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.646294 (+ 17457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.654985 (+ 8691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:43.655955 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:43.660995 (+ 5040us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:43.661348 (+ 353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:20:43.661358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:20:43.661367 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:20:43.661405 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:20:43.672678 (+ 11273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.674083 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.674089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.674093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.674273 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:20:43.674878 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:20:43.674883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:20:43.674917 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:20:43.675283 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:20:43.676976 (+ 1693us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:43.677117 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:20:43.679804 (+ 2687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":16903,"thread_start_us":16380,"threads_started":1}
I20260927 22:20:43.680588 14786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:20:43.666954 14544 master_runner.cc:186] Error getting master registration for 127.23.173.125:37475: OK, instance_id { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" instance_seqno: 1790547642876473 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:20:43.682385 14544 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:20:43.691082 14685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: Bootstrap starting.
I20260927 22:20:43.696925 14685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: Neither blocks nor log segments found. Creating new log.
I20260927 22:20:43.698235 14685 log.cc:824] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: Log is configured to *not* fsync() on all Append() calls
I20260927 22:20:43.702291 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d95e93b0be4641d3b34318f4f617c41b. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.702615 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.712661 14797 catalog_manager.cc:1347] Generated new cluster ID: 7bce0e75aa4b4e738b9eeb8acf7c1435
I20260927 22:20:43.712807 14797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:20:43.714793 14781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.715103 14781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:43.715442 14805 mvcc.cc:205] Tried to move back new op lower bound from 7334083148495593472 to 7334083148038860800. Current Snapshot: MvccSnapshot[applied={T|T < 7334083148495593472 or (T in {7334083148495593472})}]
I20260927 22:20:43.724305 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.724545 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.725615 14685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: No bootstrap required, opened a new log
I20260927 22:20:43.739773 14797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:20:43.743916 14797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:20:43.749130 14685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.749624 14685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8d8ca8e1a54ab693686cc32240b725, State: Initialized, Role: FOLLOWER
I20260927 22:20:43.750280 14685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } }
I20260927 22:20:43.771646 14685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:20:43.765368 14811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.773448 14811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.773988 14659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:20:43.773041 14660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d95e93b0be4641d3b34318f4f617c41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b8d8ca8e1a54ab693686cc32240b725"
I20260927 22:20:43.784780 14797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b: Generated new TSK 0
I20260927 22:20:43.786062 14797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:20:43.788477 14659 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [term 1 FOLLOWER]: Refusing update from remote peer d95e93b0be4641d3b34318f4f617c41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:20:43.801092 14660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d95e93b0be4641d3b34318f4f617c41b in term 1.
I20260927 22:20:43.804175 14781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:20:43.835035 14824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.835457 14824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.835801 14811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d95e93b0be4641d3b34318f4f617c41b. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:43.836030 14811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:20:43.869451 14828 catalog_manager.cc:1259] Loaded cluster ID: 7bce0e75aa4b4e738b9eeb8acf7c1435
I20260927 22:20:43.869652 14828 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: loading cluster ID for follower catalog manager: success
I20260927 22:20:43.873898 14828 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: acquiring CA information for follower catalog manager: success
I20260927 22:20:43.876551 14828 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b8d8ca8e1a54ab693686cc32240b725: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:20:43.953814 14764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:43.927874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.23.173.126:44509)
0927 22:20:43.928081 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.928090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.950174 (+ 22084us) server_negotiation.cc:407] Connection header received
0927 22:20:43.951948 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.951955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.952069 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.952155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.953492 (+ 1337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60456: BlockingRecv error: recv got EOF from 127.0.0.1:60456 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:20:43.959008 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:43.934543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.23.173.124:45435)
0927 22:20:43.934776 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.934784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.956140 (+ 21356us) server_negotiation.cc:407] Connection header received
0927 22:20:43.957356 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.957363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.957450 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.957525 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.958766 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57562: BlockingRecv error: recv got EOF from 127.0.0.1:57562 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 22:20:43.961232 14765 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:43.946747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.23.173.125:37475)
0927 22:20:43.946944 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.946950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.956715 (+ 9765us) server_negotiation.cc:407] Connection header received
0927 22:20:43.959629 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:43.959636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:43.959748 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:43.959832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:43.960958 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33348: BlockingRecv error: recv got EOF from 127.0.0.1:33348 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:44.008700 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:43.992856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.23.173.125:37475)
0927 22:20:43.993076 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:43.993084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:43.999994 (+ 6910us) server_negotiation.cc:407] Connection header received
0927 22:20:44.000639 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.000645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.000900 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.001055 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.001864 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.003067 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.005226 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.006026 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.006134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.007016 (+ 882us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.007808 (+ 792us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.007932 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.008197 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.008296 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 22:20:44.052736 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.042821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.23.173.124:45435)
0927 22:20:44.043029 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.043040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.050535 (+ 7495us) server_negotiation.cc:407] Connection header received
0927 22:20:44.051142 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.051148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.051270 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.051346 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.052457 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57576: BlockingRecv error: recv got EOF from 127.0.0.1:57576 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:20:44.100934 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.084243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.23.173.125:37475)
0927 22:20:44.084442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.084449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.094015 (+ 9566us) server_negotiation.cc:407] Connection header received
0927 22:20:44.094517 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.094524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.094639 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.094707 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.095344 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.096493 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.098633 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.099365 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.099477 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.100092 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.100233 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.100292 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.100507 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.100603 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:20:44.131946 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.128250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.23.173.124:45435)
0927 22:20:44.128495 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.128504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.130531 (+ 2027us) server_negotiation.cc:407] Connection header received
0927 22:20:44.130764 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.130770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.130855 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.130968 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.131724 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57586: BlockingRecv error: recv got EOF from 127.0.0.1:57586 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:20:44.147557 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.136596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.23.173.125:37475)
0927 22:20:44.136837 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.136846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.142521 (+ 5675us) server_negotiation.cc:407] Connection header received
0927 22:20:44.142753 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.142763 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.142849 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.142967 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.143418 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.144192 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.145680 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.146284 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.146431 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.146818 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.146939 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.147042 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.147205 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.147283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:20:44.155916 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.153657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.23.173.124:45435)
0927 22:20:44.153875 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.153882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.154550 (+ 668us) server_negotiation.cc:407] Connection header received
0927 22:20:44.154732 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.154737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.154872 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.155007 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.155734 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57596: BlockingRecv error: recv got EOF from 127.0.0.1:57596 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:20:44.169134 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.163387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.23.173.125:37475)
0927 22:20:44.163616 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.163622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.164383 (+ 761us) server_negotiation.cc:407] Connection header received
0927 22:20:44.164600 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.164606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.164687 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.164790 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.165219 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.166042 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.167351 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.167812 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.167954 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.168365 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.168500 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.168619 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.168757 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.168852 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:20:44.181852 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.179536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.23.173.124:45435)
0927 22:20:44.179846 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.179856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.180412 (+ 556us) server_negotiation.cc:407] Connection header received
0927 22:20:44.180691 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.180697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.180791 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.180909 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.181652 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57610: BlockingRecv error: recv got EOF from 127.0.0.1:57610 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 22:20:44.199750 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.193746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.23.173.125:37475)
0927 22:20:44.194022 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.194032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.194576 (+ 544us) server_negotiation.cc:407] Connection header received
0927 22:20:44.194803 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.194808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.194890 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.194992 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.195420 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.196229 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.197634 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.198267 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.198392 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.198941 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.199044 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.199149 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.199297 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.199385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 22:20:44.217100 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.214880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.23.173.124:45435)
0927 22:20:44.215128 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.215137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.215719 (+ 582us) server_negotiation.cc:407] Connection header received
0927 22:20:44.215922 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.215928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.216041 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.216157 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.216921 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57618: BlockingRecv error: recv got EOF from 127.0.0.1:57618 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:20:44.243031 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.236911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.23.173.125:37475)
0927 22:20:44.237202 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.237212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.237778 (+ 566us) server_negotiation.cc:407] Connection header received
0927 22:20:44.238016 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.238024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.238132 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.238226 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.238670 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.239610 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.241179 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.241636 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.241792 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.242236 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.242366 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.242465 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.242653 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.242738 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 22:20:44.251163 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.249040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.23.173.124:45435)
0927 22:20:44.249265 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.249273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.249794 (+ 521us) server_negotiation.cc:407] Connection header received
0927 22:20:44.250024 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.250030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.250117 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.250239 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.250975 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57632: BlockingRecv error: recv got EOF from 127.0.0.1:57632 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:20:44.293048 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.287371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.23.173.125:37475)
0927 22:20:44.287616 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.287625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.288172 (+ 547us) server_negotiation.cc:407] Connection header received
0927 22:20:44.288422 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.288426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.288496 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.288605 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.289028 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.289827 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.291176 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.291617 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.291770 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.292166 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.292298 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.292395 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.292541 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.292618 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 22:20:44.301016 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.299137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.23.173.124:45435)
0927 22:20:44.299325 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.299334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.299692 (+ 358us) server_negotiation.cc:407] Connection header received
0927 22:20:44.299901 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.299906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.299991 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.300095 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.300822 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57640: BlockingRecv error: recv got EOF from 127.0.0.1:57640 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:20:44.375134 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.369080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.23.173.125:37475)
0927 22:20:44.369311 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.369317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.369664 (+ 347us) server_negotiation.cc:407] Connection header received
0927 22:20:44.369904 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.369917 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.370039 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.370146 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.370552 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.371434 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.373067 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.373683 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.373842 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.374258 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.374446 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.374534 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.374767 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.374850 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:20:44.401859 14803 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:20:44.399627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.23.173.124:45435)
0927 22:20:44.399810 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.399817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.400360 (+ 543us) server_negotiation.cc:407] Connection header received
0927 22:20:44.400539 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.400545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.400643 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.400753 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:20:44.401636 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57646: BlockingRecv error: recv got EOF from 127.0.0.1:57646 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:20:44.411158 14781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d95e93b0be4641d3b34318f4f617c41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95e93b0be4641d3b34318f4f617c41b" member_type: VOTER last_known_addr { host: "127.23.173.126" port: 44509 } } peers { permanent_uuid: "8b8d8ca8e1a54ab693686cc32240b725" member_type: VOTER last_known_addr { host: "127.23.173.125" port: 37475 } } peers { permanent_uuid: "09c3167713284fcb8f63b9121c7ee619" member_type: VOTER last_known_addr { host: "127.23.173.124" port: 45435 } } }
I20260927 22:20:44.411571 14781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95e93b0be4641d3b34318f4f617c41b [sys.catalog]: This master's current role is: LEADER
I20260927 22:20:44.483615 14778 catalog_manager.cc:1259] Loaded cluster ID: 7bce0e75aa4b4e738b9eeb8acf7c1435
I20260927 22:20:44.483749 14778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: loading cluster ID for follower catalog manager: success
I20260927 22:20:44.486452 14778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: acquiring CA information for follower catalog manager: success
I20260927 22:20:44.488369 14778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09c3167713284fcb8f63b9121c7ee619: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:20:44.538719 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.533722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.23.173.125:37475)
0927 22:20:44.533983 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.533990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.534307 (+ 317us) server_negotiation.cc:407] Connection header received
0927 22:20:44.534469 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.534473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.534544 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.534626 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.535013 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.535708 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.537028 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.537458 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.537599 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.538029 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.538127 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.538229 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.538380 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.538460 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:20:44.549329 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.543803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.23.173.124:45435)
0927 22:20:44.543960 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.543964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.544405 (+ 441us) server_negotiation.cc:407] Connection header received
0927 22:20:44.544535 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.544539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.544714 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.544812 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.545131 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.545921 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.547294 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.547698 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.547788 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.548232 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.548736 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.548853 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.548944 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.549008 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:20:44.573114 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.555007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.23.173.126:44509)
0927 22:20:44.566438 (+ 11431us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.566442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.566460 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:20:44.566561 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.566565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.566730 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.566932 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.567605 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.568780 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.570274 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.570893 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.571017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.571859 (+ 842us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.572324 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.572394 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.572675 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.572770 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11326,"thread_start_us":11199,"threads_started":1}
I20260927 22:20:44.573859 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.556204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.23.173.124:45435)
0927 22:20:44.556427 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.556435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.567051 (+ 10616us) server_negotiation.cc:407] Connection header received
0927 22:20:44.567399 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.567408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.567515 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.567583 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.568374 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.569380 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.571697 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.572389 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.572557 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.573056 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.573204 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.573283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.573487 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.573592 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:20:44.573946 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:20:44.555201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.23.173.125:37475)
0927 22:20:44.555425 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:20:44.555432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:20:44.567846 (+ 12414us) server_negotiation.cc:407] Connection header received
0927 22:20:44.567978 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:20:44.567983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:20:44.568082 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:20:44.568190 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:20:44.568783 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.569815 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.571749 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:20:44.572414 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:20:44.572557 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:20:44.573172 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:20:44.573308 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:20:44.573368 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:20:44.573574 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:20:44.573648 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:20:44.578876 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14544
I20260927 22:20:44.579921 14606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:44.690785 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14544
I20260927 22:20:44.712006 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14618
I20260927 22:20:44.713127 14680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:44.821295 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14618
I20260927 22:20:44.841490 24245 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu pid 14692
I20260927 22:20:44.842700 14754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:20:44.949831 24245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskO6uUAX/build/asan/bin/kudu with pid 14692
2026-09-27T22:20:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4384 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9515 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361667 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST